Oct 30 22:04:03.544737 ip-10-0-135-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 22:04:03.544751 ip-10-0-135-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 22:04:03.544759 ip-10-0-135-132 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 22:04:03.545105 ip-10-0-135-132 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 22:04:13.672739 ip-10-0-135-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 22:04:13.672759 ip-10-0-135-132 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c59ef6ca1df84ea4a5e50a96b04b60db --
Oct 30 22:06:13.130240 ip-10-0-135-132 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:06:13.551061 ip-10-0-135-132 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:13.551061 ip-10-0-135-132 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:06:13.551061 ip-10-0-135-132 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:13.551061 ip-10-0-135-132 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:13.554506 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554423    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.554506 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554506    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.554506 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554510    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554513    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554517    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554520    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554523    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554525    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554528    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554531    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554534    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554536    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554540    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554544    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554548    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554550    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554553    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554571    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554576    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554579    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554583    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.554603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554586    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554589    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554591    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554599    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554609    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554614    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554617    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554620    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554623    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554625    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554628    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554631    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554634    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554636    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554639    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554641    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554643    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554646    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554649    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554651    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.555067 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554653    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554656    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554659    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554661    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554663    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554667    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554669    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554671    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554674    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554676    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554679    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554682    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554684    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554687    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554690    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554692    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554695    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554698    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554700    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554703    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.555530 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554705    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554708    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554710    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554712    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554715    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554717    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554720    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554722    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554725    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554727    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554729    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554732    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554734    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554736    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554739    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554741    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554744    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554746    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554749    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554751    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.556006 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554754    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554770    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554774    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554776    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554779    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554783    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554787    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554791    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554795    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554798    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554801    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554804    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554806    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554809    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554812    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554814    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554817    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554819    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554822    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.556469 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554825    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554827    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554829    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554832    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554835    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.554837    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555594    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555600    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555603    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555606    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555608    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555612    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555614    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555617    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555619    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555623    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555626    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555628    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555632    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555634    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.556932 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555637    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555640    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555642    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555646    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555648    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555651    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555654    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555656    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555659    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555661    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555664    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555666    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555669    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555671    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555674    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555677    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555679    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555682    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555686    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.557450 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555689    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555692    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555695    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555697    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555700    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555702    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555705    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555707    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555710    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555712    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555715    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555718    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555721    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555723    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555726    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555728    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555730    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555733    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555736    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.557941 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555738    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555741    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555743    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555746    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555748    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555751    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555754    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555772    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555775    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555777    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555780    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555782    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555785    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555788    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555791    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555793    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555796    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555798    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555801    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555803    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.559101 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555806    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555809    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555811    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555814    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555818    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555821    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555823    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555826    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555828    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555831    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555834    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555837    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555840    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555842    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555845    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555847    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555850    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555852    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555855    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.559594 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555857    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555861    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555865    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555868    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555870    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555873    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555876    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555878    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555880    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555883    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555885    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555888    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555890    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555892    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.555897    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556387    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556391    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556395    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556398    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.560059 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556400    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556403    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556405    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556408    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556410    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556414    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556417    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556420    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556423    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556427    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556430    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556433    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556436    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556439    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556441    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556444    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556447    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556450    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556452    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.560515 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556455    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556457    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556460    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556462    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556465    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556467    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556470    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556472    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556475    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556478    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556481    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556483    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556486    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556488    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556491    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556493    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556496    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556498    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556500    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556504    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.560978 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556507    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556509    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556511    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556514    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556516    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556518    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556521    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556523    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556526    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556528    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556531    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556533    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556536    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556538    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556540    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556543    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556546    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556549    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556551    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556554    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.561435 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556556    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556559    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556562    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556567    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556570    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556572    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556574    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556577    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556580    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556582    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556584    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556587    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556590    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556592    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556595    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556597    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556599    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556602    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556605    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556607    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.561924 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556610    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556612    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556615    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556617    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556620    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556622    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556625    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556627    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556630    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556632    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556635    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556637    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556641    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556645    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556647    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556652    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556654    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556657    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556659    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556662    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.562438 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556664    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556668    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.556670    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557491    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557505    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557513    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557518    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557523    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557526    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557531    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557535    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557539    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557542    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557545    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557549    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557552    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557571    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557575    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557579    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557582    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557585    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557590    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557593    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557596    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:06:13.562925 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557599    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557602    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557606    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557611    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557614    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557617    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557620    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557623    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557626    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557629    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557632    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557637    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557640    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557643    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557649    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557652    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557655    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557660    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557663    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557666    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557668    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557671    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557677    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557680    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557683    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:06:13.563482 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557686    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557689    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557692    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557695    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557698    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557701    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557711    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557715    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557719    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557722    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557726    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557729    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557733    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557736    2576 flags.go:64] FLAG: --help="false"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557739    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557742    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557745    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557748    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557751    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557754    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557771    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557774    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557778    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:06:13.564203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557781    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557784    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557787    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557791    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557794    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557797    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557802    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557805    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557808    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557811    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557814    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557817    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557823    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557826    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557829    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557832    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557835    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557839    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557842    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557846    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557850    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557853    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557857    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557860    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557867    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:06:13.564742 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557871    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557874    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557877    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557880    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557883    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557890    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557893    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557896    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557900    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557902    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557905    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557908    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557911    2576 flags.go:64] FLAG: --port="10250"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557916    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557919    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fd78c64da7cc6732"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557922    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557925    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557928    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557931    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557935    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557937    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557941    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557944    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557948    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557951    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557954    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:06:13.565354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557958    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557961    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557964    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557967    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557970    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557973    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557976    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557979    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557982    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557985    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557988    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557991    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557994    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.557997    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558001    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558004    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558007    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558012    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558015    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558020    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558024    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558026    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558029    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558032    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558035    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:06:13.566020 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558038    2576 flags.go:64] FLAG: --v="2"
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558043    2576 flags.go:64] FLAG: --version="false"
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558048    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558056    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558059    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558230    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558234    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558238    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558241    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558243    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558246    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558248    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558251    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558254    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558256    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558259    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558261    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558264    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558266    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558269    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558271    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.566603 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558274    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558277    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558279    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558282    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558284    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558289    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558293    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558296    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558299    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558303    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558306    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558309    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558311    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558314    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558316    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558319    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558321    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558324    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558327    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.567124 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558329    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558332    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558334    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558338    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558342    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558344    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558347    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558349    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558352    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558354    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558357    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558360    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558362    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558364    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558367    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558370    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558373    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558375    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558379    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.567577 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558382    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558384    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558386    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558389    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558391    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558394    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558396    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558399    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558401    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558403    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558406    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558408    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558411    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558413    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558416    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558418    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558421    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558423    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558426    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558428    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.568053 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558430    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558433    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558435    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558438    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558440    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558442    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558445    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558447    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558450    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558452    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558455    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558458    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558461    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558463    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558466    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558468    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558471    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558473    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558476    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.568513 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558478    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558481    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558483    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558485    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558502    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558505    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558507    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558510    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558513    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558515    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558518    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558520    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558523    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.558550    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558588    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558592    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.569255 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558595    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558598    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558600    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558603    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558605    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558608    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558610    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558613    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558617    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558623    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558626    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558629    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558631    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558634    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558636    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558639    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558641    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558643    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558646    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.569895 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558649    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558652    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558655    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558657    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558660    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558662    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558666    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558669    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558672    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558674    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558677    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558679    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558681    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558684    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558687    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558690    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558692    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558695    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558698    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.570405 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558700    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558702    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558706    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558710    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558713    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558715    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558718    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558720    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558722    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558725    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558727    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558730    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558732    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558734    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558738    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558741    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558743    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558745    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558748    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558750    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.570876 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558753    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558755    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558772    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558774    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558777    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558779    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558782    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558784    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558787    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558789    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558792    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558795    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558797    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558800    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558802    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558807    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558809    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558812    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558815    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.571352 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558817    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558820    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558822    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558825    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558827    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558829    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558832    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558837    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558839    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558842    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558844    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558847    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558849    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558852    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558854    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558857    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558860    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558862    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558865    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558867    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.571845 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558870    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558872    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558875    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558878    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558881    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558883    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.558886    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.566199    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.566222    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.566748    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.566754    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566840    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566848    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566850    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566854    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566857    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566860    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566862    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566865    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566867    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.572316 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566870    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566872    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566874    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566877    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566879    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566882    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566884    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566886    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566889    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566891    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566895    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566897    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566899    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566902    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566904    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566907    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566909    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566912    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566915    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566918    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.572900 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566920    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566930    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566933    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566936    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566938    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566940    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566943    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566945    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566947    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566950    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566952    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566955    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566957    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566959    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566962    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566964    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566966    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566969    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566972    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566974    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.573369 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566976    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566978    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566981    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566983    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566986    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566988    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566991    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566993    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566995    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.566998    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567001    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567003    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567006    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567008    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567016    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567019    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567021    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567024    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567027    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.573866 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567030    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567034    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567037    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567040    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567042    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567045    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567048    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567050    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567053    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567055    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567057    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567060    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567062    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567065    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567067    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567070    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567072    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567074    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.574311 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567077    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567080    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567082    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567085    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567087    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567090    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567093    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567095    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567097    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567101    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567111    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567115    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567118    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567120    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567123    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567126    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567128    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567131    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567133    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.574747 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567136    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.567167    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567376    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567381    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567384    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567386    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567389    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567391    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567394    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567396    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567399    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567401    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567404    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567406    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567410    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.575219 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567414    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567417    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567420    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567423    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567425    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567428    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567430    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567433    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567435    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567443    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567447    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567449    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567451    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567454    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567456    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567459    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567461    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567463    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567466    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567468    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.575586 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567470    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567473    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567475    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567478    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567480    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567483    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567485    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567488    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567490    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567493    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567496    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567498    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567500    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567503    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567506    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567508    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567511    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567513    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567516    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.576111 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567518    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567521    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567523    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567531    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567534    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567536    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567539    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567541    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567544    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567546    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567548    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567551    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567553    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567555    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567558    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567560    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567563    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567565    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567567    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567570    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.576565 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567572    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567575    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567577    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567580    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567582    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567585    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567587    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567591    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567595    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567598    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567601    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567604    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567612    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567615    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567618    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567620    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567630    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567634    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567636    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.577043 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567638    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567641    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567643    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567645    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567648    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567650    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567652    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567655    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567657    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567659    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567662    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567665    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567667    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567670    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567672    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.567695    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:13.577481 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567730    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567735    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567738    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567740    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567742    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567746    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567750    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567753    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567767    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567771    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567773    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567776    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567778    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567781    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567790    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567793    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567795    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567797    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567800    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:13.577870 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567802    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567805    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567807    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567809    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567812    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567814    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567816    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567819    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567821    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567823    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567826    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567828    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567830    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567833    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567835    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567838    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567841    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567843    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567846    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567848    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:13.578344 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567850    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567853    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567855    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567857    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567860    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567863    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567865    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567867    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567875    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567878    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567880    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567883    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567885    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567887    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567890    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567892    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567894    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567897    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567899    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567901    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:13.578861 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567904    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567907    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567912    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567915    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567918    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567920    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567923    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567925    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567929    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567932    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567934    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567937    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567939    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567949    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567953    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567957    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567959    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567962    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567965    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:13.579455 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567967    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567970    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567978    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567980    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567983    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567986    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567988    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567990    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567993    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567995    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.567998    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568000    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568002    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568005    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568007    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568011    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568014    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568016    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568019    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:13.579918 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568021    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568024    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568027    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568030    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568032    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568035    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568038    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568040    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:13.568043    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.568094    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.571468    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.572494    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.572585    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.572616    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.574540    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:13.580502 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.575685    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:13.586402 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.586382    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:06:13.589644 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.589629    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 22:06:13.590472 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.590444    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:06:13.594294 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.594274    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8325bcd5-ce36-42cd-b331-bc78295d5e01:/dev/nvme0n1p4 90fab635-ac8b-41e6-9721-b7562f79524b:/dev/nvme0n1p3]
Oct 30 22:06:13.594356 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.594292    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 22:06:13.600723 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.600594    2576 manager.go:217] Machine: {Timestamp:2026-10-30 22:06:13.599338218 +0000 UTC m=+0.347338911 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127887 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e6b413026cb8178cded090f9d37c8 SystemUUID:ec2e6b41-3026-cb81-78cd-ed090f9d37c8 BootID:c59ef6ca-1df8-4ea4-a5e5-0a96b04b60db Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c3:d3:f5:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c3:d3:f5:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:df:79:89:c5:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:06:13.600823 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.600724    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:06:13.600879 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.600866    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:06:13.602031 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.601998    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:06:13.602181 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.602038    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:06:13.602679 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.602670    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:06:13.602708 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.602682    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:06:13.602708 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.602701    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:13.602780 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.602712    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:13.603535 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.603524    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:06:13.607268 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.607254    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:06:13.607391 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.607382    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:06:13.608223 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.608213    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:06:13.608318 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.608237    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:06:13.608318 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.608257    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:06:13.608318 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.608269    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:06:13.608318 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.608280    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:06:13.608827 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.608812    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:13.613235 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.613212    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:06:13.617952 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.617939    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:13.618729 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.618717    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:06:13.618783 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.618747    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:06:13.618915 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.618905    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:06:13.618944 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.618925    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:06:13.618944 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.618935    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:06:13.618944 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.618942    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:06:13.619025 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.618948    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:06:13.619025 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.618953    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:06:13.619653 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.619645    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:06:13.619682 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.619653    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:06:13.619682 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.619661    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:06:13.619682 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.619667    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:06:13.619682 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.619676    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:06:13.619823 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.619685    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:06:13.620616 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.620607    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:06:13.620649 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.620616    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:06:13.624450 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.624438    2576 server.go:1263] "Started kubelet"
Oct 30 22:06:13.624549 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.624494    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:06:13.624658 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.624548    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:06:13.624658 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.624649    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:06:13.625248 ip-10-0-135-132 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:06:13.627301 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.627288    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:06:13.628477 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.628456    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:06:13.630265 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.630246    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 22:06:13.632451 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.632432    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:13.633271 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.633251    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:06:13.633347 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.633333    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:13.634520 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.634491    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:06:13.634520 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.634508    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:06:13.634654 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.634596    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:06:13.634702 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.634664    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:06:13.634702 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.634673    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:06:13.634901 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.634873    2576 factory.go:55] Registering systemd factory
Oct 30 22:06:13.634901 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.634894    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:06:13.635265 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.635248    2576 factory.go:153] Registering CRI-O factory
Oct 30 22:06:13.635326 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.635270    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:06:13.635361 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.635338    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:06:13.635413 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.635367    2576 factory.go:103] Registering Raw factory
Oct 30 22:06:13.635413 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.635396    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:06:13.636307 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.636291    2576 manager.go:319] Starting recovery of all containers
Oct 30 22:06:13.648598 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.648571    2576 manager.go:324] Recovery completed
Oct 30 22:06:13.653716 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.653694    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.655717 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.655691    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 22:06:13.655828 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.655693    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 22:06:13.656706 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.655702    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-132.ec2.internal.18e36df486c9cf59  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-132.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-132.ec2.internal,},FirstTimestamp:2026-10-30 22:06:13.624409945 +0000 UTC m=+0.372410640,LastTimestamp:2026-10-30 22:06:13.624409945 +0000 UTC m=+0.372410640,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-132.ec2.internal,}"
Oct 30 22:06:13.657193 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.657168    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.657291 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.657196    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.657291 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.657206    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.657892 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.657877    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:06:13.657892 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.657892    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:06:13.657970 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.657879    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.657970 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.657912    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:06:13.658753 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.658734    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.658811 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.658790    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.658811 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.658805    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.660235 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.660223    2576 policy_none.go:50] "Start"
Oct 30 22:06:13.660659 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.660650    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:06:13.660687 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.660663    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:06:13.662724 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.662713    2576 policy_none.go:44] "Start"
Oct 30 22:06:13.666073 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.666056    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:06:13.666144 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.666118    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hj72t"
Oct 30 22:06:13.670822 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.670795    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 22:06:13.670911 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.670833    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 22:06:13.670986 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.670962    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 22:06:13.673944 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.673929    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hj72t"
Oct 30 22:06:13.686930 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.686740    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-132.ec2.internal.18e36df488bde258  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-132.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-132.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-132.ec2.internal,},FirstTimestamp:2026-10-30 22:06:13.657182808 +0000 UTC m=+0.405183501,LastTimestamp:2026-10-30 22:06:13.657182808 +0000 UTC m=+0.405183501,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-132.ec2.internal,}"
Oct 30 22:06:13.693403 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.693370    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:06:13.695628 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.695615    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:06:13.696125 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.696104    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:06:13.696178 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.696130    2576 server.go:87] "Starting device plugin registration server"
Oct 30 22:06:13.696419 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.696406    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:06:13.696455 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.696423    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:06:13.696540 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.696524    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:06:13.696636 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.696620    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:06:13.696636 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.696633    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:06:13.697166 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.697147    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:06:13.697244 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.697193    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:13.770140 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.770073    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:06:13.772112 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.771412    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:06:13.772112 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.771433    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:06:13.772112 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.771453    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:06:13.772112 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.771494    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 22:06:13.787843 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.787815    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:13.796870 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.796854    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.797780 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.797746    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.797879 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.797800    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.797879 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.797813    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.797879 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.797837    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.818629 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.818607    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.818717 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.818633    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-132.ec2.internal\": node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:13.860803 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.860754    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:13.871583 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.871555    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"]
Oct 30 22:06:13.871699 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.871624    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.872519 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.872505    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.872598 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.872537    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.872598 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.872554    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.874783 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.874752    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.874917 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.874902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.874961 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.874932    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.875574 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.875557    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.875636 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.875587    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.875636 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.875597    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.875636 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.875557    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.875782 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.875658    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.875782 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.875670    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.877677 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.877662    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.877725 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.877690    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:13.878433 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.878415    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:13.878530 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.878442    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:13.878530 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.878456    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:13.902247 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.902227    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-132.ec2.internal\" not found" node="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.905088 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.905071    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-132.ec2.internal\" not found" node="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.936124 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.936098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f5710f91ddba841cbff091b973dc1da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-132.ec2.internal\" (UID: \"3f5710f91ddba841cbff091b973dc1da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.936228 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.936126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.936228 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:13.936143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:13.961603 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:13.961570    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.036590 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.036526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:14.036590 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.036555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:14.036590 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.036573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f5710f91ddba841cbff091b973dc1da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-132.ec2.internal\" (UID: \"3f5710f91ddba841cbff091b973dc1da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:14.036801 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.036599    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f5710f91ddba841cbff091b973dc1da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-132.ec2.internal\" (UID: \"3f5710f91ddba841cbff091b973dc1da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:14.036801 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.036636    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:14.036801 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.036637    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:14.062575 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.062535    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.163574 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.163534    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.205750 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.205715    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:14.208887 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.208864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:14.264643 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.264609    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.365211 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.365140    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.465737 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.465699    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.566235 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.566201    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.572364 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.572341    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 22:06:14.572491 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.572461    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 22:06:14.667088 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.667010    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.678999 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.678970    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:01:15 +0000 UTC" deadline="2028-07-26 17:53:21.579208074 +0000 UTC"
Oct 30 22:06:14.678999 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.678993    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15235h47m6.900217789s"
Oct 30 22:06:14.767646 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.767610    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.864725 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:14.864690    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7605b2d536800c86463888458ce09c5.slice/crio-04d9f44dea51a8df5db2e7029036128e537ed479d898f6d84aee7c0610bff3fa WatchSource:0}: Error finding container 04d9f44dea51a8df5db2e7029036128e537ed479d898f6d84aee7c0610bff3fa: Status 404 returned error can't find the container with id 04d9f44dea51a8df5db2e7029036128e537ed479d898f6d84aee7c0610bff3fa
Oct 30 22:06:14.864909 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.864804    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:14.866053 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.866036    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:14.866142 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.866068    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:14.866142 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.866078    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:14.867681 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.867665    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:14.869468 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.869451    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:06:14.894873 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.894844    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:14.894984 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:14.894884    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:14.967941 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:14.967877    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:15.010138 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:15.010115    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:15.034103 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:15.034083    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:15.057639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:15.057618    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:15.058662 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:15.058648    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:15.077247 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:15.077223    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:17.190342 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.190275    2576 apiserver.go:52] "Watching apiserver"
Oct 30 22:06:17.214936 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.214905    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:17.234560 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.234529    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:06:17.234830 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.234811    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ndn7k","openshift-multus/network-metrics-daemon-xb5xn","openshift-ovn-kubernetes/ovnkube-node-kqbrc","kube-system/konnectivity-agent-vgcnn","kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7","openshift-cluster-node-tuning-operator/tuned-pb69k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal","openshift-multus/multus-additional-cni-plugins-876md","openshift-network-operator/iptables-alerter-8kmq8","openshift-image-registry/node-ca-6zmgp","openshift-multus/multus-l5shw","openshift-network-diagnostics/network-check-target-2k9k2","kube-system/global-pull-secret-syncer-lcd4g"]
Oct 30 22:06:17.237611 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.237582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.239807 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.239787    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.242286 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.242269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.244693 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.244666    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:17.247011 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.246992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.247104 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.247066    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:17.249276 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.249255    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.251706 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.251685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.254280 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.254261    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g5ww4\""
Oct 30 22:06:17.254381 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.254350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:17.254442 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.254405    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:17.254506 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.254492    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.254556 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.254501    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.255004 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.254988    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xwfpf\""
Oct 30 22:06:17.255152 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.255124    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:06:17.255395 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.255365    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.255467 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.255425    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.256432 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.256413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:17.256511 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.256482    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:17.259077 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.259059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.261287 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.261266    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.263538 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.263517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.277104 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.277083    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:06:17.277292 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.277137    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rvj6h"
Oct 30 22:06:17.307075 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.307046    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:17.317524 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.317363    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:06:17.326963 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.326932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-var-lib-kubelet\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.327104 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.326971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c86d61-12bf-4446-b98a-f9e571ad5ffb-host\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.327104 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.326998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-netd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.327104 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-device-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.327104 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-systemd-units\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnrc\" (UniqueName: \"kubernetes.io/projected/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kube-api-access-rqnrc\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df124a61-d687-4ef9-a732-2d5def6ef79c-hosts-file\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-var-lib-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqnf\" (UniqueName: \"kubernetes.io/projected/7f07db13-af88-4f66-a3ea-23b4314d6144-kube-api-access-mvqnf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.327348 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-netns\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-conf-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-modprobe-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-log-socket\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-script-lib\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df124a61-d687-4ef9-a732-2d5def6ef79c-tmp-dir\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnpjh\" (UniqueName: \"kubernetes.io/projected/df124a61-d687-4ef9-a732-2d5def6ef79c-kube-api-access-fnpjh\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-tuned\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-config\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-dbus\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10654728-36ab-4482-9fcb-775d88ab5aa0-host-slash\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327599    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdd6d\" (UniqueName: \"kubernetes.io/projected/543555d6-bbb3-4643-8198-9c7c397c07d9-kube-api-access-tdd6d\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10654728-36ab-4482-9fcb-775d88ab5aa0-iptables-alerter-script\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.327767 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327655    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-cni-binary-copy\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-kubernetes\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-run\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-socket-dir-parent\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-bin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85f74\" (UniqueName: \"kubernetes.io/projected/dece32b6-cb70-479d-9a12-bc2608b23129-kube-api-access-85f74\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-conf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-slash\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-netns\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-node-log\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-k8s-cni-cncf-io\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysconfig\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-sys\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.328543 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.327987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-kubelet\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91c86d61-12bf-4446-b98a-f9e571ad5ffb-serviceca\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc4fx\" (UniqueName: \"kubernetes.io/projected/91c86d61-12bf-4446-b98a-f9e571ad5ffb-kube-api-access-qc4fx\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-etc-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxhf\" (UniqueName: \"kubernetes.io/projected/8b91ef29-ba61-4265-a756-ac5ad6ab604d-kube-api-access-ckxhf\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-systemd\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7665c55-0b43-4318-9bad-342fce7a359d-konnectivity-ca\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-system-cni-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cnibin\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-host\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-kubelet\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovn-node-metrics-cert\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-system-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.329299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-cnibin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-multus\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-hostroot\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-bin\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fbd\" (UniqueName: \"kubernetes.io/projected/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-kube-api-access-k8fbd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-etc-kubernetes\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-tmp\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-ovn\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-env-overrides\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgr5b\" (UniqueName: \"kubernetes.io/projected/10654728-36ab-4482-9fcb-775d88ab5aa0-kube-api-access-bgr5b\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-multus-daemon-config\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-systemd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7665c55-0b43-4318-9bad-342fce7a359d-agent-certs\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:17.329810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-kubelet-config\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.330342 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-binary-copy\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.330342 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-multus-certs\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.330342 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.328698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-lib-modules\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.354395 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.354362    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:06:17.354554 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.354526    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:06:17.355908 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.355879    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:06:17.356057 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356028    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tjmjw\""
Oct 30 22:06:17.356165 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356093    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.356165 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356144    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.356304 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356210    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:06:17.356304 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356269    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.356399 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356336    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.356399 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356381    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356627    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pwrc4\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356675    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356763    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356836    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.356918    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.357020    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.357094    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.357266    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.357445    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.357524    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.357576    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:06:17.358002 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.357617    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:06:17.360850 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.360804    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal" event={"ID":"3f5710f91ddba841cbff091b973dc1da","Type":"ContainerStarted","Data":"4eb7586174bc6b86c522d8f2b9e84000ec80a3849b6f217acf7459ce2fd9980d"}
Oct 30 22:06:17.362176 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.362149    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal" event={"ID":"d7605b2d536800c86463888458ce09c5","Type":"ContainerStarted","Data":"04d9f44dea51a8df5db2e7029036128e537ed479d898f6d84aee7c0610bff3fa"}
Oct 30 22:06:17.369202 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.368964    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:06:17.369202 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.368977    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7kw8g\""
Oct 30 22:06:17.369202 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.369038    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:06:17.369202 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.369098    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8wxtm\""
Oct 30 22:06:17.369202 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.369130    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44l7l\""
Oct 30 22:06:17.369202 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.369134    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k6z4j\""
Oct 30 22:06:17.369202 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.369171    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn9g8\""
Oct 30 22:06:17.429280 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-multus\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.429280 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-hostroot\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-bin\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-hostroot\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fbd\" (UniqueName: \"kubernetes.io/projected/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-kube-api-access-k8fbd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-etc-kubernetes\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429439    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-tmp\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-multus\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-bin\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.429514 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-ovn\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429531    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-ovn\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429613    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-etc-kubernetes\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-env-overrides\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgr5b\" (UniqueName: \"kubernetes.io/projected/10654728-36ab-4482-9fcb-775d88ab5aa0-kube-api-access-bgr5b\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-multus-daemon-config\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-systemd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7665c55-0b43-4318-9bad-342fce7a359d-agent-certs\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-kubelet-config\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-binary-copy\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-multus-certs\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-lib-modules\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-var-lib-kubelet\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c86d61-12bf-4446-b98a-f9e571ad5ffb-host\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-netd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430098 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-device-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-systemd-units\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnrc\" (UniqueName: \"kubernetes.io/projected/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kube-api-access-rqnrc\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df124a61-d687-4ef9-a732-2d5def6ef79c-hosts-file\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-var-lib-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqnf\" (UniqueName: \"kubernetes.io/projected/7f07db13-af88-4f66-a3ea-23b4314d6144-kube-api-access-mvqnf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-netns\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430514    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-multus-daemon-config\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-conf-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430552    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-env-overrides\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-modprobe-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-log-socket\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430616    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-systemd-units\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.430851 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-log-socket\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-multus-certs\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-script-lib\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430676    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-var-lib-kubelet\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df124a61-d687-4ef9-a732-2d5def6ef79c-tmp-dir\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430710    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df124a61-d687-4ef9-a732-2d5def6ef79c-hosts-file\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430780    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-var-lib-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430805    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-lib-modules\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.430727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnpjh\" (UniqueName: \"kubernetes.io/projected/df124a61-d687-4ef9-a732-2d5def6ef79c-kube-api-access-fnpjh\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.429831    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.430897    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431168    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-systemd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431183    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431207    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431275    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-netd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-tuned\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431353    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-kubelet-config\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.431367    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret podName:099dbe30-a9b3-4166-b289-7fc8e25ff414 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:17.931347109 +0000 UTC m=+3.097891688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret") pod "global-pull-secret-syncer-lcd4g" (UID: "099dbe30-a9b3-4166-b289-7fc8e25ff414") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:17.431639 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-config\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431454    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-device-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431484    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-netns\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c86d61-12bf-4446-b98a-f9e571ad5ffb-host\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431678    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-modprobe-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-dbus\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431775    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-conf-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10654728-36ab-4482-9fcb-775d88ab5aa0-host-slash\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdd6d\" (UniqueName: \"kubernetes.io/projected/543555d6-bbb3-4643-8198-9c7c397c07d9-kube-api-access-tdd6d\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10654728-36ab-4482-9fcb-775d88ab5aa0-iptables-alerter-script\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-cni-binary-copy\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.431994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-kubernetes\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-script-lib\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-run\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.432481 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432485    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10654728-36ab-4482-9fcb-775d88ab5aa0-host-slash\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-socket-dir-parent\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-bin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.432674    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85f74\" (UniqueName: \"kubernetes.io/projected/dece32b6-cb70-479d-9a12-bc2608b23129-kube-api-access-85f74\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432558    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-kubernetes\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.432831    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:17.932740615 +0000 UTC m=+3.099285185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432885    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432854    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-dbus\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432931    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.432987    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-socket-dir-parent\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433151    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-bin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433191    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-binary-copy\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.433270 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433268    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-run\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433304    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433490    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-conf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-slash\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-netns\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-node-log\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-k8s-cni-cncf-io\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysconfig\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-sys\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.433933 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-kubelet\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91c86d61-12bf-4446-b98a-f9e571ad5ffb-serviceca\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.433982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4fx\" (UniqueName: \"kubernetes.io/projected/91c86d61-12bf-4446-b98a-f9e571ad5ffb-kube-api-access-qc4fx\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-etc-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxhf\" (UniqueName: \"kubernetes.io/projected/8b91ef29-ba61-4265-a756-ac5ad6ab604d-kube-api-access-ckxhf\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-systemd\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7665c55-0b43-4318-9bad-342fce7a359d-konnectivity-ca\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-system-cni-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cnibin\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434181    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-cni-binary-copy\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-host\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-kubelet\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovn-node-metrics-cert\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-system-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-cnibin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434460    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-cnibin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434495    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-etc-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.434537 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434533    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-sys\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-netns\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434653    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-conf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434684    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-slash\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434713    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-node-log\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434764    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-k8s-cni-cncf-io\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-systemd\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysconfig\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434871    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.435288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.434938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435397    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-kubelet\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435464    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-host\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435465    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cnibin\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435487    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-kubelet\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435508    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-system-cni-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-tuned\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-system-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.435711 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435699    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-tmp\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.436116 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91c86d61-12bf-4446-b98a-f9e571ad5ffb-serviceca\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.436116 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.435984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.436116 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.436021    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7665c55-0b43-4318-9bad-342fce7a359d-agent-certs\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:17.436480 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.436435    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10654728-36ab-4482-9fcb-775d88ab5aa0-iptables-alerter-script\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.436593 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.436505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df124a61-d687-4ef9-a732-2d5def6ef79c-tmp-dir\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.436593 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.436530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-config\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.437370 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.437302    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7665c55-0b43-4318-9bad-342fce7a359d-konnectivity-ca\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:17.438288 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.438269    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovn-node-metrics-cert\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.452609 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.452552    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:17.452609 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.452577    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:17.452609 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.452593    2576 projected.go:196] Error preparing data for projected volume kube-api-access-r4r4g for pod openshift-network-diagnostics/network-check-target-2k9k2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:17.452821 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.452668    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g podName:650a07ef-80ee-44a8-8e96-b6bd104beb37 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:17.952646589 +0000 UTC m=+3.119191163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r4r4g" (UniqueName: "kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g") pod "network-check-target-2k9k2" (UID: "650a07ef-80ee-44a8-8e96-b6bd104beb37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:17.454046 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.454014    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fbd\" (UniqueName: \"kubernetes.io/projected/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-kube-api-access-k8fbd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.457042 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.457016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqnf\" (UniqueName: \"kubernetes.io/projected/7f07db13-af88-4f66-a3ea-23b4314d6144-kube-api-access-mvqnf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.457445 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.457422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnpjh\" (UniqueName: \"kubernetes.io/projected/df124a61-d687-4ef9-a732-2d5def6ef79c-kube-api-access-fnpjh\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.458743 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.458720    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdd6d\" (UniqueName: \"kubernetes.io/projected/543555d6-bbb3-4643-8198-9c7c397c07d9-kube-api-access-tdd6d\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:17.459743 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.459687    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnrc\" (UniqueName: \"kubernetes.io/projected/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kube-api-access-rqnrc\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.460488 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.460101    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgr5b\" (UniqueName: \"kubernetes.io/projected/10654728-36ab-4482-9fcb-775d88ab5aa0-kube-api-access-bgr5b\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.460488 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.460294    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxhf\" (UniqueName: \"kubernetes.io/projected/8b91ef29-ba61-4265-a756-ac5ad6ab604d-kube-api-access-ckxhf\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.462260 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.462228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4fx\" (UniqueName: \"kubernetes.io/projected/91c86d61-12bf-4446-b98a-f9e571ad5ffb-kube-api-access-qc4fx\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.463321 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.463298    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85f74\" (UniqueName: \"kubernetes.io/projected/dece32b6-cb70-479d-9a12-bc2608b23129-kube-api-access-85f74\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.551362 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.551332    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:17.560684 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.560658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:17.562651 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.562619    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f07db13_af88_4f66_a3ea_23b4314d6144.slice/crio-9f2fcaf5e4f664273eb2222c796dd9cb079084f8c33623bf2c326cbc62246ee3 WatchSource:0}: Error finding container 9f2fcaf5e4f664273eb2222c796dd9cb079084f8c33623bf2c326cbc62246ee3: Status 404 returned error can't find the container with id 9f2fcaf5e4f664273eb2222c796dd9cb079084f8c33623bf2c326cbc62246ee3
Oct 30 22:06:17.571498 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.571453    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c86d61_12bf_4446_b98a_f9e571ad5ffb.slice/crio-af13edebc8f85b896b61c7a475e84342ca69c450d793816b00bebbe13e0a9b35 WatchSource:0}: Error finding container af13edebc8f85b896b61c7a475e84342ca69c450d793816b00bebbe13e0a9b35: Status 404 returned error can't find the container with id af13edebc8f85b896b61c7a475e84342ca69c450d793816b00bebbe13e0a9b35
Oct 30 22:06:17.585429 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.585404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:17.588672 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.588645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:17.593340 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.593173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:17.597207 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.597178    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7665c55_0b43_4318_9bad_342fce7a359d.slice/crio-3978efc022e99c0ed1477c13e15b2665f5ba7b5ff9f9bfb9f6ec7d231e74c88f WatchSource:0}: Error finding container 3978efc022e99c0ed1477c13e15b2665f5ba7b5ff9f9bfb9f6ec7d231e74c88f: Status 404 returned error can't find the container with id 3978efc022e99c0ed1477c13e15b2665f5ba7b5ff9f9bfb9f6ec7d231e74c88f
Oct 30 22:06:17.602511 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.602485    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9995b8_ba05_47ba_81c6_6f74c234f2c1.slice/crio-7a75252d36590d5bfc165a9678835be2ac5f2a31183b5b1962b1ec4fcaaff560 WatchSource:0}: Error finding container 7a75252d36590d5bfc165a9678835be2ac5f2a31183b5b1962b1ec4fcaaff560: Status 404 returned error can't find the container with id 7a75252d36590d5bfc165a9678835be2ac5f2a31183b5b1962b1ec4fcaaff560
Oct 30 22:06:17.605784 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.604991    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99488cc5_1b6c_4b64_a38e_6403a080f2a1.slice/crio-3ef9ef4c440384a67fd36c991c0814e7c1b29a5ff4761cda9e2f51a171c0b7b7 WatchSource:0}: Error finding container 3ef9ef4c440384a67fd36c991c0814e7c1b29a5ff4761cda9e2f51a171c0b7b7: Status 404 returned error can't find the container with id 3ef9ef4c440384a67fd36c991c0814e7c1b29a5ff4761cda9e2f51a171c0b7b7
Oct 30 22:06:17.613360 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.613335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:17.615486 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.615463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:17.616937 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.616919    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:17.623305 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.623211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5shw"
Oct 30 22:06:17.629373 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.629338    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b91ef29_ba61_4265_a756_ac5ad6ab604d.slice/crio-9a2a70f900401dfa0c683d458fb33c586efa7eda71e387ef35f684a1d75a0074 WatchSource:0}: Error finding container 9a2a70f900401dfa0c683d458fb33c586efa7eda71e387ef35f684a1d75a0074: Status 404 returned error can't find the container with id 9a2a70f900401dfa0c683d458fb33c586efa7eda71e387ef35f684a1d75a0074
Oct 30 22:06:17.630825 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.630667    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf124a61_d687_4ef9_a732_2d5def6ef79c.slice/crio-59adf563b1f1cc53ffcd5dcaae43b926565cad753f1bb334086797eed8a8c65c WatchSource:0}: Error finding container 59adf563b1f1cc53ffcd5dcaae43b926565cad753f1bb334086797eed8a8c65c: Status 404 returned error can't find the container with id 59adf563b1f1cc53ffcd5dcaae43b926565cad753f1bb334086797eed8a8c65c
Oct 30 22:06:17.636051 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.636023    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10654728_36ab_4482_9fcb_775d88ab5aa0.slice/crio-b744483fcd9cd00c46423a3d59c46e07919a6ff9e8e38a37226dec01ee572ab6 WatchSource:0}: Error finding container b744483fcd9cd00c46423a3d59c46e07919a6ff9e8e38a37226dec01ee572ab6: Status 404 returned error can't find the container with id b744483fcd9cd00c46423a3d59c46e07919a6ff9e8e38a37226dec01ee572ab6
Oct 30 22:06:17.637033 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:17.637009    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddece32b6_cb70_479d_9a12_bc2608b23129.slice/crio-2abf750bbc0db41984c5c318d945be8bc50daabe03cace872f1c43ba72be5a44 WatchSource:0}: Error finding container 2abf750bbc0db41984c5c318d945be8bc50daabe03cace872f1c43ba72be5a44: Status 404 returned error can't find the container with id 2abf750bbc0db41984c5c318d945be8bc50daabe03cace872f1c43ba72be5a44
Oct 30 22:06:17.937864 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.937624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:17.938036 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:17.937905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:17.938036 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.937797    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:17.938036 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.938015    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret podName:099dbe30-a9b3-4166-b289-7fc8e25ff414 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:18.937994608 +0000 UTC m=+4.104539175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret") pod "global-pull-secret-syncer-lcd4g" (UID: "099dbe30-a9b3-4166-b289-7fc8e25ff414") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:17.938209 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.938067    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:17.938209 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:17.938131    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:18.938107526 +0000 UTC m=+4.104652092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:18.038290 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.038265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:18.038474 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.038456    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:18.038543 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.038492    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:18.038543 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.038506    2576 projected.go:196] Error preparing data for projected volume kube-api-access-r4r4g for pod openshift-network-diagnostics/network-check-target-2k9k2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:18.038643 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.038560    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g podName:650a07ef-80ee-44a8-8e96-b6bd104beb37 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:19.038541162 +0000 UTC m=+4.205085715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4r4g" (UniqueName: "kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g") pod "network-check-target-2k9k2" (UID: "650a07ef-80ee-44a8-8e96-b6bd104beb37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:18.280794 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.280686    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:17 +0000 UTC" deadline="2028-06-03 08:01:26.225600386 +0000 UTC"
Oct 30 22:06:18.280794 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.280749    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13953h55m7.94485474s"
Oct 30 22:06:18.353701 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.353619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:18.353849 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.353764    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:18.371469 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.371433    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5shw" event={"ID":"dece32b6-cb70-479d-9a12-bc2608b23129","Type":"ContainerStarted","Data":"2abf750bbc0db41984c5c318d945be8bc50daabe03cace872f1c43ba72be5a44"}
Oct 30 22:06:18.373769 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.373739    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerStarted","Data":"9a2a70f900401dfa0c683d458fb33c586efa7eda71e387ef35f684a1d75a0074"}
Oct 30 22:06:18.382791 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.382760    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7" event={"ID":"99488cc5-1b6c-4b64-a38e-6403a080f2a1","Type":"ContainerStarted","Data":"3ef9ef4c440384a67fd36c991c0814e7c1b29a5ff4761cda9e2f51a171c0b7b7"}
Oct 30 22:06:18.389004 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.388969    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6zmgp" event={"ID":"91c86d61-12bf-4446-b98a-f9e571ad5ffb","Type":"ContainerStarted","Data":"af13edebc8f85b896b61c7a475e84342ca69c450d793816b00bebbe13e0a9b35"}
Oct 30 22:06:18.397819 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.397784    2576 generic.go:356] "Generic (PLEG): container finished" podID="d7605b2d536800c86463888458ce09c5" containerID="a883995867f6b4dab01af83bc8d2b5a9e960df40397dcb2b97b44cfb7c297f7d" exitCode=0
Oct 30 22:06:18.397955 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.397872    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal" event={"ID":"d7605b2d536800c86463888458ce09c5","Type":"ContainerDied","Data":"a883995867f6b4dab01af83bc8d2b5a9e960df40397dcb2b97b44cfb7c297f7d"}
Oct 30 22:06:18.411677 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.411642    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kmq8" event={"ID":"10654728-36ab-4482-9fcb-775d88ab5aa0","Type":"ContainerStarted","Data":"b744483fcd9cd00c46423a3d59c46e07919a6ff9e8e38a37226dec01ee572ab6"}
Oct 30 22:06:18.422279 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.422233    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ndn7k" event={"ID":"df124a61-d687-4ef9-a732-2d5def6ef79c","Type":"ContainerStarted","Data":"59adf563b1f1cc53ffcd5dcaae43b926565cad753f1bb334086797eed8a8c65c"}
Oct 30 22:06:18.442844 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.442803    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"7a75252d36590d5bfc165a9678835be2ac5f2a31183b5b1962b1ec4fcaaff560"}
Oct 30 22:06:18.461306 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.461264    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgcnn" event={"ID":"c7665c55-0b43-4318-9bad-342fce7a359d","Type":"ContainerStarted","Data":"3978efc022e99c0ed1477c13e15b2665f5ba7b5ff9f9bfb9f6ec7d231e74c88f"}
Oct 30 22:06:18.486755 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.486643    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pb69k" event={"ID":"7f07db13-af88-4f66-a3ea-23b4314d6144","Type":"ContainerStarted","Data":"9f2fcaf5e4f664273eb2222c796dd9cb079084f8c33623bf2c326cbc62246ee3"}
Oct 30 22:06:18.510480 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.509775    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal" event={"ID":"3f5710f91ddba841cbff091b973dc1da","Type":"ContainerStarted","Data":"baf484fd55e32ecbabd2eedfdf411b21854c741975e73ed62ff6b41d5fe4ae77"}
Oct 30 22:06:18.551705 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.551631    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal" podStartSLOduration=2.551609381 podStartE2EDuration="2.551609381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:18.550266731 +0000 UTC m=+3.716811316" watchObservedRunningTime="2026-10-30 22:06:18.551609381 +0000 UTC m=+3.718153957"
Oct 30 22:06:18.950298 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.949335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:18.950298 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:18.949393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:18.950298 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.949559    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:18.950298 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.949732    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:20.94969907 +0000 UTC m=+6.116243628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:18.950298 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.950135    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:18.950298 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:18.950188    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret podName:099dbe30-a9b3-4166-b289-7fc8e25ff414 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:20.950171561 +0000 UTC m=+6.116716118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret") pod "global-pull-secret-syncer-lcd4g" (UID: "099dbe30-a9b3-4166-b289-7fc8e25ff414") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:19.050622 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:19.050000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:19.050622 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:19.050158    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:19.050622 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:19.050181    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:19.050622 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:19.050194    2576 projected.go:196] Error preparing data for projected volume kube-api-access-r4r4g for pod openshift-network-diagnostics/network-check-target-2k9k2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:19.050622 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:19.050266    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g podName:650a07ef-80ee-44a8-8e96-b6bd104beb37 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:21.050249066 +0000 UTC m=+6.216793635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4r4g" (UniqueName: "kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g") pod "network-check-target-2k9k2" (UID: "650a07ef-80ee-44a8-8e96-b6bd104beb37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:19.281951 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:19.281809    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:17 +0000 UTC" deadline="2028-08-15 20:31:09.305487 +0000 UTC"
Oct 30 22:06:19.281951 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:19.281869    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15718h24m50.023621382s"
Oct 30 22:06:19.356113 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:19.355407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:19.356113 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:19.355541    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:19.356113 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:19.355939    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:19.356113 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:19.356028    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:19.523390 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:19.523284    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal" event={"ID":"d7605b2d536800c86463888458ce09c5","Type":"ContainerStarted","Data":"261b87bc22d1c812cf3583657b33c99fdf5931d36d10a6ec009fc7e58d6a7125"}
Oct 30 22:06:19.563506 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:19.562985    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal" podStartSLOduration=3.562967138 podStartE2EDuration="3.562967138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:19.562410199 +0000 UTC m=+4.728954773" watchObservedRunningTime="2026-10-30 22:06:19.562967138 +0000 UTC m=+4.729511715"
Oct 30 22:06:20.353450 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:20.353400    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:20.353898 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:20.353536    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:20.532174 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:20.531973    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="bbf8617d8500207a12c2243b9df4b3ba11f0fc3b2b3021739a00d55bc92ebc33" exitCode=0
Oct 30 22:06:20.532174 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:20.532121    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerDied","Data":"bbf8617d8500207a12c2243b9df4b3ba11f0fc3b2b3021739a00d55bc92ebc33"}
Oct 30 22:06:20.964470 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:20.964423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:20.964658 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:20.964537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:20.964720 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:20.964662    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:20.964765 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:20.964726    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret podName:099dbe30-a9b3-4166-b289-7fc8e25ff414 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:24.964706723 +0000 UTC m=+10.131251289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret") pod "global-pull-secret-syncer-lcd4g" (UID: "099dbe30-a9b3-4166-b289-7fc8e25ff414") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:20.965150 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:20.965110    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:20.965252 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:20.965203    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:24.965188238 +0000 UTC m=+10.131732813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:21.066171 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:21.065576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:21.066171 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:21.065716    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:21.066171 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:21.065734    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:21.066171 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:21.065746    2576 projected.go:196] Error preparing data for projected volume kube-api-access-r4r4g for pod openshift-network-diagnostics/network-check-target-2k9k2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:21.066171 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:21.065805    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g podName:650a07ef-80ee-44a8-8e96-b6bd104beb37 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:25.065785527 +0000 UTC m=+10.232330104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4r4g" (UniqueName: "kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g") pod "network-check-target-2k9k2" (UID: "650a07ef-80ee-44a8-8e96-b6bd104beb37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:21.362566 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:21.361911    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:21.362566 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:21.362112    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:21.362566 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:21.362323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:21.362566 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:21.362503    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:21.540647 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:21.540472    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgcnn" event={"ID":"c7665c55-0b43-4318-9bad-342fce7a359d","Type":"ContainerStarted","Data":"e556363727369f18966a7d55036b64b9398892a8593a309b4e7b8c1d6f20fa8e"}
Oct 30 22:06:22.354090 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:22.354054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:22.354286 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:22.354194    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:23.092930 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:23.092894    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:23.093448 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:23.093420    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:23.132884 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:23.132838    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vgcnn" podStartSLOduration=4.54396468 podStartE2EDuration="8.132825236s" totalImagesPullingTime="3.588860556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:21.58278873 +0000 UTC m=+6.749333305" watchObservedRunningTime="2026-10-30 22:06:23.132825236 +0000 UTC m=+8.299369810"
Oct 30 22:06:23.358844 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:23.358131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:23.358844 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:23.358285    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:23.358844 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:23.358649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:23.358844 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:23.358741    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:23.554938 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:23.554209    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6zmgp" event={"ID":"91c86d61-12bf-4446-b98a-f9e571ad5ffb","Type":"ContainerStarted","Data":"6b6e6e11205e39a37461c906d55c1730da27df6630595e5671da8e3630a1a7f2"}
Oct 30 22:06:23.558283 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:23.558252    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7" event={"ID":"99488cc5-1b6c-4b64-a38e-6403a080f2a1","Type":"ContainerStarted","Data":"db0f3097715fbf67c442bd92b09943b1b34ec695f5cc81cb268aad6064a0b501"}
Oct 30 22:06:24.353799 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:24.353764    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:24.354312 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:24.353918    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:24.561664 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:24.560742    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:06:24.787377 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:24.787350    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:06:25.001657 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.000843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:25.001657 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.000897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:25.001657 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.001053    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:25.001657 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.001114    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:33.00109667 +0000 UTC m=+18.167641259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:25.001657 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.001529    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:25.001657 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.001577    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret podName:099dbe30-a9b3-4166-b289-7fc8e25ff414 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:33.001562351 +0000 UTC m=+18.168106917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret") pod "global-pull-secret-syncer-lcd4g" (UID: "099dbe30-a9b3-4166-b289-7fc8e25ff414") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:25.101726 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.101683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:25.101916 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.101900    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:25.101983 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.101924    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:25.101983 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.101937    2576 projected.go:196] Error preparing data for projected volume kube-api-access-r4r4g for pod openshift-network-diagnostics/network-check-target-2k9k2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:25.102081 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.101994    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g podName:650a07ef-80ee-44a8-8e96-b6bd104beb37 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:33.101976567 +0000 UTC m=+18.268521118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4r4g" (UniqueName: "kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g") pod "network-check-target-2k9k2" (UID: "650a07ef-80ee-44a8-8e96-b6bd104beb37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:25.284114 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.283937    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:06:24.787370893Z","UUID":"d8fccba6-5bcc-4fb8-8dae-b3bbdb00e1c5","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:06:25.297319 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.296973    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:06:25.297319 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.297011    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:06:25.356620 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.356597    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:25.356998 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.356711    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:25.356998 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.356748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:25.356998 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:25.356837    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:25.565899 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.565646    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5shw" event={"ID":"dece32b6-cb70-479d-9a12-bc2608b23129","Type":"ContainerStarted","Data":"d11bf41ae3348dbbbf9b55346607b1ae43d8490f47caabf15e0f90f8f1bbf58e"}
Oct 30 22:06:25.571748 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.571712    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="d2c26e55e5cd8bc1f1910376c0c0f13b13d8ccc6b3809b9d0ec7b99c4f102f6d" exitCode=0
Oct 30 22:06:25.571909 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.571798    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerDied","Data":"d2c26e55e5cd8bc1f1910376c0c0f13b13d8ccc6b3809b9d0ec7b99c4f102f6d"}
Oct 30 22:06:25.576778 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.576704    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7" event={"ID":"99488cc5-1b6c-4b64-a38e-6403a080f2a1","Type":"ContainerStarted","Data":"f1c1ddc6794a1a10b19512eecf1cffad1f65886e79cb5b564c7387f1465b9dd0"}
Oct 30 22:06:25.578747 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.578708    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ndn7k" event={"ID":"df124a61-d687-4ef9-a732-2d5def6ef79c","Type":"ContainerStarted","Data":"721bc4bba7242c567d04ee4cae2f2fbbb67e88558c588886eac842a8f61998b7"}
Oct 30 22:06:25.608511 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.608454    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6zmgp" podStartSLOduration=5.597761278 podStartE2EDuration="10.608437445s" totalImagesPullingTime="5.010676167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:23.595266043 +0000 UTC m=+8.761810618" watchObservedRunningTime="2026-10-30 22:06:25.608437445 +0000 UTC m=+10.774982020"
Oct 30 22:06:25.608688 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:25.608559    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-l5shw" podStartSLOduration=2.885812198 podStartE2EDuration="10.608553933s" totalImagesPullingTime="7.722741735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:25.608421725 +0000 UTC m=+10.774966301" watchObservedRunningTime="2026-10-30 22:06:25.608553933 +0000 UTC m=+10.775098508"
Oct 30 22:06:26.354065 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:26.354038    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:26.354193 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:26.354163    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:26.584505 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:26.584416    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="5faaf6c7d3ccacd9e477dc0638ed89cac71e992c47e6136508b19bb8c7639b5d" exitCode=0
Oct 30 22:06:26.584936 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:26.584525    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerDied","Data":"5faaf6c7d3ccacd9e477dc0638ed89cac71e992c47e6136508b19bb8c7639b5d"}
Oct 30 22:06:26.588141 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:26.587873    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7" event={"ID":"99488cc5-1b6c-4b64-a38e-6403a080f2a1","Type":"ContainerStarted","Data":"f140766b086838b0e540b9b7b45324bdf9591519dfee123bc88ce774fbcd4f47"}
Oct 30 22:06:26.630093 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:26.630032    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ndn7k" podStartSLOduration=4.041040124 podStartE2EDuration="11.630014663s" totalImagesPullingTime="7.588974539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:25.695257924 +0000 UTC m=+10.861802494" watchObservedRunningTime="2026-10-30 22:06:26.630014663 +0000 UTC m=+11.796559240"
Oct 30 22:06:26.673089 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:26.673039    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7" podStartSLOduration=3.710774122 podStartE2EDuration="11.673024555s" totalImagesPullingTime="7.962250433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:26.672388585 +0000 UTC m=+11.838933158" watchObservedRunningTime="2026-10-30 22:06:26.673024555 +0000 UTC m=+11.839569129"
Oct 30 22:06:27.353960 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:27.353785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:27.353960 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:27.353891    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:27.354314 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:27.354294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:27.354490 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:27.354400    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:27.591649 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:27.591549    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kmq8" event={"ID":"10654728-36ab-4482-9fcb-775d88ab5aa0","Type":"ContainerStarted","Data":"3f688e1e4de26581ee13e6c1310e00c22fd6cf043a72a18e4d28ca90c2f8027b"}
Oct 30 22:06:27.594435 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:27.594407    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="96726f20f4c01f736c3ada0d7e11311434081ad6143c963e1478483b990698ee" exitCode=0
Oct 30 22:06:27.594580 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:27.594495    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerDied","Data":"96726f20f4c01f736c3ada0d7e11311434081ad6143c963e1478483b990698ee"}
Oct 30 22:06:27.631775 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:27.631719    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8kmq8" podStartSLOduration=4.836814662 podStartE2EDuration="12.631703448s" totalImagesPullingTime="7.794888786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:27.63126391 +0000 UTC m=+12.797808489" watchObservedRunningTime="2026-10-30 22:06:27.631703448 +0000 UTC m=+12.798248022"
Oct 30 22:06:28.353698 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:28.353668    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:28.353882 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:28.353801    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:28.600150 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:28.600106    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pb69k" event={"ID":"7f07db13-af88-4f66-a3ea-23b4314d6144","Type":"ContainerStarted","Data":"6247d1872783f3c0f3872cb4a62ba3f6ed91e1afc3b809185e6f56b529d964fd"}
Oct 30 22:06:28.643898 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:28.643773    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pb69k" podStartSLOduration=3.554942772 podStartE2EDuration="13.643754604s" totalImagesPullingTime="10.088811832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:28.642652854 +0000 UTC m=+13.809197430" watchObservedRunningTime="2026-10-30 22:06:28.643754604 +0000 UTC m=+13.810299180"
Oct 30 22:06:29.354163 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:29.354126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:29.354353 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:29.354297    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:29.354415 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:29.354351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:29.354448 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:29.354435    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:29.604903 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:29.604828    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerStarted","Data":"e3ed1ed24427a46ffb3685bb8ff495e6344df7aa6286e3169bbad37782937c80"}
Oct 30 22:06:30.353854 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:30.353812    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:30.354024 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:30.353954    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:30.610863 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:30.610569    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="e3ed1ed24427a46ffb3685bb8ff495e6344df7aa6286e3169bbad37782937c80" exitCode=0
Oct 30 22:06:30.610863 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:30.610648    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerDied","Data":"e3ed1ed24427a46ffb3685bb8ff495e6344df7aa6286e3169bbad37782937c80"}
Oct 30 22:06:31.353764 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:31.353728    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:31.353764 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:31.353763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:31.353996 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:31.353865    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:31.354062 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:31.354035    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:31.616116 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:31.616035    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="6cd2fc344130ad7ef77f228d73a274255e86016f6eb4ba459ea3a3f088e6722e" exitCode=0
Oct 30 22:06:31.616116 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:31.616088    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerDied","Data":"6cd2fc344130ad7ef77f228d73a274255e86016f6eb4ba459ea3a3f088e6722e"}
Oct 30 22:06:32.354137 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.353915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:32.354296 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:32.354181    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:32.624322 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.624228    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-876md" event={"ID":"8b91ef29-ba61-4265-a756-ac5ad6ab604d","Type":"ContainerStarted","Data":"17f532307d7f3ed5ef7015bf57aa0ca9cecd444b9cf48c203df4d868aa3c0e27"}
Oct 30 22:06:32.626927 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.626906    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:06:32.627263 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.627245    2576 generic.go:356] "Generic (PLEG): container finished" podID="ad9995b8-ba05-47ba-81c6-6f74c234f2c1" containerID="e0aff1a9d7e122e6e880aba12f8d7e9e920822bae4e96f27d08978f8b911071b" exitCode=1
Oct 30 22:06:32.627325 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.627281    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"23d7ab2c756008af1377c7645635b9382e55615cbe28a644e267d9d080663e40"}
Oct 30 22:06:32.627359 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.627328    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"e2e7d231830796400e2115922c1807c5fbcc2fed7b07f9257a3bb5e009d1a156"}
Oct 30 22:06:32.627403 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.627358    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"f21577be2d3449895dd6900931b232e1d3576c77a81b87f6c03f351697ed9fa8"}
Oct 30 22:06:32.627403 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.627386    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"811319741fc09fe60c085840549f7d192364992704689f1811e62ba76f2c92ce"}
Oct 30 22:06:32.627461 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.627412    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerDied","Data":"e0aff1a9d7e122e6e880aba12f8d7e9e920822bae4e96f27d08978f8b911071b"}
Oct 30 22:06:32.627461 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.627440    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"c05df42c8bcf937f15de7cf4b597e68181e013dff1405bb155f9e9959b818b15"}
Oct 30 22:06:32.698277 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:32.698195    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-876md" podStartSLOduration=7.536449924 podStartE2EDuration="17.698180742s" totalImagesPullingTime="10.161730818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:32.697699623 +0000 UTC m=+17.864244188" watchObservedRunningTime="2026-10-30 22:06:32.698180742 +0000 UTC m=+17.864725315"
Oct 30 22:06:33.066783 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:33.066697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:33.066783 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:33.066757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:33.066988 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.066844    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:33.066988 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.066881    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:33.066988 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.066905    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret podName:099dbe30-a9b3-4166-b289-7fc8e25ff414 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:49.066890307 +0000 UTC m=+34.233434859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret") pod "global-pull-secret-syncer-lcd4g" (UID: "099dbe30-a9b3-4166-b289-7fc8e25ff414") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:33.066988 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.066932    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:49.066914961 +0000 UTC m=+34.233459526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:33.167549 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:33.167507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:33.167687 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.167634    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:33.167687 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.167649    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:33.167687 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.167657    2576 projected.go:196] Error preparing data for projected volume kube-api-access-r4r4g for pod openshift-network-diagnostics/network-check-target-2k9k2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:33.167789 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.167706    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g podName:650a07ef-80ee-44a8-8e96-b6bd104beb37 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:49.16769329 +0000 UTC m=+34.334237843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4r4g" (UniqueName: "kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g") pod "network-check-target-2k9k2" (UID: "650a07ef-80ee-44a8-8e96-b6bd104beb37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:33.353844 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:33.353774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:33.353974 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.353884    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:33.353974 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:33.353914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:33.353974 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:33.353958    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:34.353367 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:34.353328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:34.353818 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:34.353434    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:34.634813 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:34.634741    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:06:34.635080 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:34.635060    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"70d9c7fe88919b21853c0b67c599ee3ca806dc5448bd18a25ecd698eaed0cdce"}
Oct 30 22:06:35.354963 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:35.354924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:35.355347 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:35.355057    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:35.355347 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:35.355104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:35.355347 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:35.355203    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:36.353491 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:36.353441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:36.353655 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:36.353578    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:36.642617 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:36.642539    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:06:36.643405 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:36.642870    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"e5ed4822f9e23a61f843e12f7463d1fdea517d6d899d40fa330da0ab16193f14"}
Oct 30 22:06:36.643405 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:36.643170    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:36.643405 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:36.643251    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:36.643405 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:36.643274    2576 scope.go:122] "RemoveContainer" containerID="e0aff1a9d7e122e6e880aba12f8d7e9e920822bae4e96f27d08978f8b911071b"
Oct 30 22:06:36.660330 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:36.660308    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:37.353695 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:37.353506    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:37.353967 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:37.353519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:37.353967 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:37.353804    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:37.353967 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:37.353874    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:37.658150 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:37.658012    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:06:37.659819 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:37.659761    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" event={"ID":"ad9995b8-ba05-47ba-81c6-6f74c234f2c1","Type":"ContainerStarted","Data":"9ae0735dd81e7a4bfd53a3815770d798a30daad4a20d65e679db2472406dca6c"}
Oct 30 22:06:37.660844 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:37.660825    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:37.678164 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:37.678138    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:37.749547 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:37.749500    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc" podStartSLOduration=8.618049728999999 podStartE2EDuration="22.749486683s" totalImagesPullingTime="14.131436954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:37.749295473 +0000 UTC m=+22.915840046" watchObservedRunningTime="2026-10-30 22:06:37.749486683 +0000 UTC m=+22.916031249"
Oct 30 22:06:38.353415 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:38.353349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:38.353581 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:38.353459    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:38.399106 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:38.399062    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lcd4g"]
Oct 30 22:06:38.402200 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:38.402176    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2k9k2"]
Oct 30 22:06:38.402354 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:38.402297    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:38.402412 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:38.402374    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:38.402935 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:38.402911    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xb5xn"]
Oct 30 22:06:38.403056 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:38.403013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:38.403124 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:38.403108    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:38.661970 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:38.661894    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:38.662369 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:38.662015    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:40.353150 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:40.353114    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:40.353628 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:40.353165    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:40.353628 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:40.353283    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:40.353628 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:40.353291    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:40.353628 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:40.353379    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:40.353628 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:40.353445    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:42.353556 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:42.353523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:42.353556 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:42.353555    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:42.354053 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:42.353556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:42.354053 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:42.353649    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2k9k2" podUID="650a07ef-80ee-44a8-8e96-b6bd104beb37"
Oct 30 22:06:42.354053 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:42.353762    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xb5xn" podUID="543555d6-bbb3-4643-8198-9c7c397c07d9"
Oct 30 22:06:42.354053 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:42.353821    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lcd4g" podUID="099dbe30-a9b3-4166-b289-7fc8e25ff414"
Oct 30 22:06:44.052901 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.052824    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeReady"
Oct 30 22:06:44.053253 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.052960    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 22:06:44.121626 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.121596    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-456zn"]
Oct 30 22:06:44.146984 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.146954    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tjbxb"]
Oct 30 22:06:44.148203 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.148167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:44.154233 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.154195    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-456zn"]
Oct 30 22:06:44.154350 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.154252    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tjbxb"]
Oct 30 22:06:44.154350 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.154315    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.164062 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.164039    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:06:44.164062 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.164055    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-htsvp\""
Oct 30 22:06:44.164599 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.164586    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:06:44.164669 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.164586    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:06:44.239732 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.239692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gbgm\" (UniqueName: \"kubernetes.io/projected/16934604-e78f-4968-bdcd-3e48bd4f6882-kube-api-access-5gbgm\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:44.239884 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.239744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdr8r\" (UniqueName: \"kubernetes.io/projected/ec7d111b-c9ab-455f-acdd-a016d4f48531-kube-api-access-bdr8r\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.239884 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.239847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7d111b-c9ab-455f-acdd-a016d4f48531-config-volume\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.239884 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.239873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.239980 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.239965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16934604-e78f-4968-bdcd-3e48bd4f6882-cert\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:44.240039 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.240027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7d111b-c9ab-455f-acdd-a016d4f48531-tmp-dir\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.264205 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.264166    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:06:44.266909 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.264597    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z6gtt\""
Oct 30 22:06:44.267318 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.267300    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:06:44.340980 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.340947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16934604-e78f-4968-bdcd-3e48bd4f6882-cert\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:44.340980 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.340986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7d111b-c9ab-455f-acdd-a016d4f48531-tmp-dir\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.341276 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.341019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gbgm\" (UniqueName: \"kubernetes.io/projected/16934604-e78f-4968-bdcd-3e48bd4f6882-kube-api-access-5gbgm\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:44.341276 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.341041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdr8r\" (UniqueName: \"kubernetes.io/projected/ec7d111b-c9ab-455f-acdd-a016d4f48531-kube-api-access-bdr8r\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.341276 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.341069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7d111b-c9ab-455f-acdd-a016d4f48531-config-volume\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.341276 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.341089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.341276 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:44.341201    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:44.341519 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:44.341282    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls podName:ec7d111b-c9ab-455f-acdd-a016d4f48531 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:44.841263577 +0000 UTC m=+30.007808132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls") pod "dns-default-tjbxb" (UID: "ec7d111b-c9ab-455f-acdd-a016d4f48531") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:44.341816 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.341786    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7d111b-c9ab-455f-acdd-a016d4f48531-tmp-dir\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.341816 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.341814    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7d111b-c9ab-455f-acdd-a016d4f48531-config-volume\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.345519 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.345391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16934604-e78f-4968-bdcd-3e48bd4f6882-cert\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:44.354121 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.354098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:44.354299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.354131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:44.354299 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.354235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:44.361926 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.361905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdr8r\" (UniqueName: \"kubernetes.io/projected/ec7d111b-c9ab-455f-acdd-a016d4f48531-kube-api-access-bdr8r\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.362025 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.361939    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gbgm\" (UniqueName: \"kubernetes.io/projected/16934604-e78f-4968-bdcd-3e48bd4f6882-kube-api-access-5gbgm\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:44.370283 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.370260    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:06:44.370403 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.370341    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:06:44.370467 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.370448    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:06:44.370546 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.370480    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:06:44.370546 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.370497    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggpvh\""
Oct 30 22:06:44.370620 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.370609    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bf7kf\""
Oct 30 22:06:44.370710 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.370696    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5qz2\""
Oct 30 22:06:44.463365 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.463330    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:44.600854 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.600825    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:44.600980 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.600962    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:06:44.601598 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.601576    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:44.816677 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.816640    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-456zn"]
Oct 30 22:06:44.820885 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:44.820812    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16934604_e78f_4968_bdcd_3e48bd4f6882.slice/crio-47bc1a3636731a6bc141ab3e482018f455c62c071f4f4d566f2ad452c4dfe579 WatchSource:0}: Error finding container 47bc1a3636731a6bc141ab3e482018f455c62c071f4f4d566f2ad452c4dfe579: Status 404 returned error can't find the container with id 47bc1a3636731a6bc141ab3e482018f455c62c071f4f4d566f2ad452c4dfe579
Oct 30 22:06:44.845458 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:44.845422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:44.845612 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:44.845573    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:44.845646 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:44.845633    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls podName:ec7d111b-c9ab-455f-acdd-a016d4f48531 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:45.845618347 +0000 UTC m=+31.012162902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls") pod "dns-default-tjbxb" (UID: "ec7d111b-c9ab-455f-acdd-a016d4f48531") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:45.680186 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:45.680146    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-456zn" event={"ID":"16934604-e78f-4968-bdcd-3e48bd4f6882","Type":"ContainerStarted","Data":"47bc1a3636731a6bc141ab3e482018f455c62c071f4f4d566f2ad452c4dfe579"}
Oct 30 22:06:45.853432 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:45.853399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:45.853606 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:45.853584    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:45.853683 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:45.853671    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls podName:ec7d111b-c9ab-455f-acdd-a016d4f48531 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.853651734 +0000 UTC m=+33.020196310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls") pod "dns-default-tjbxb" (UID: "ec7d111b-c9ab-455f-acdd-a016d4f48531") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:47.686853 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:47.686818    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-456zn" event={"ID":"16934604-e78f-4968-bdcd-3e48bd4f6882","Type":"ContainerStarted","Data":"801c754ab67bd223de1179df0313a1fdfe8be2b191b787930f790bdac9b4b75c"}
Oct 30 22:06:47.871960 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:47.871864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:47.872105 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:47.872049    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:47.872142 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:47.872117    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls podName:ec7d111b-c9ab-455f-acdd-a016d4f48531 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.872101547 +0000 UTC m=+37.038646100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls") pod "dns-default-tjbxb" (UID: "ec7d111b-c9ab-455f-acdd-a016d4f48531") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:48.729917 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:48.729774    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-456zn" podStartSLOduration=1.9715801210000001 podStartE2EDuration="4.729757107s" totalImagesPullingTime="2.758176986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:48.729655727 +0000 UTC m=+33.896200305" watchObservedRunningTime="2026-10-30 22:06:48.729757107 +0000 UTC m=+33.896301682"
Oct 30 22:06:49.080384 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.080295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:49.080384 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.080342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:49.080568 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:49.080442    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:49.080568 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:49.080495    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:21.080481679 +0000 UTC m=+66.247026244 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : secret "metrics-daemon-secret" not found
Oct 30 22:06:49.083152 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.083126    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:49.181556 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.181524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:49.184065 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.184035    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:49.189989 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.189973    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:49.193184 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.193168    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:49.519668 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.519601    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lcd4g"]
Oct 30 22:06:49.522859 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:49.522833    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099dbe30_a9b3_4166_b289_7fc8e25ff414.slice/crio-5adcb1750e290d2acefa7779ffc18568ab27e26eb820d97a29b358ab3927c61b WatchSource:0}: Error finding container 5adcb1750e290d2acefa7779ffc18568ab27e26eb820d97a29b358ab3927c61b: Status 404 returned error can't find the container with id 5adcb1750e290d2acefa7779ffc18568ab27e26eb820d97a29b358ab3927c61b
Oct 30 22:06:49.549356 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.549321    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2k9k2"]
Oct 30 22:06:49.552538 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:49.552509    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650a07ef_80ee_44a8_8e96_b6bd104beb37.slice/crio-3fbd3473f977c340d1879e65b736833ea0ae5b94dfb876b9032aab258f9a0484 WatchSource:0}: Error finding container 3fbd3473f977c340d1879e65b736833ea0ae5b94dfb876b9032aab258f9a0484: Status 404 returned error can't find the container with id 3fbd3473f977c340d1879e65b736833ea0ae5b94dfb876b9032aab258f9a0484
Oct 30 22:06:49.694570 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.694529    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2k9k2" event={"ID":"650a07ef-80ee-44a8-8e96-b6bd104beb37","Type":"ContainerStarted","Data":"3fbd3473f977c340d1879e65b736833ea0ae5b94dfb876b9032aab258f9a0484"}
Oct 30 22:06:49.695817 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:49.695790    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lcd4g" event={"ID":"099dbe30-a9b3-4166-b289-7fc8e25ff414","Type":"ContainerStarted","Data":"5adcb1750e290d2acefa7779ffc18568ab27e26eb820d97a29b358ab3927c61b"}
Oct 30 22:06:50.092498 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.092465    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jd57c"]
Oct 30 22:06:50.110646 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.109841    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.123411 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.123381    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jd57c"]
Oct 30 22:06:50.125716 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.125674    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:06:50.135689 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.135644    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26rrv\""
Oct 30 22:06:50.151201 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.151170    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-cn27f-rccw9"]
Oct 30 22:06:50.159862 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.159826    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:06:50.159862 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.159851    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:06:50.160168 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.160150    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:06:50.164622 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.164599    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.176160 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.176137    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cn27f-rccw9"]
Oct 30 22:06:50.186727 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.186676    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vjfsq\""
Oct 30 22:06:50.187191 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.187099    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:06:50.191810 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.191788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f749f836-83b3-4266-95d7-390a8c19f113-data-volume\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.191913 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.191834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tstdw\" (UniqueName: \"kubernetes.io/projected/f749f836-83b3-4266-95d7-390a8c19f113-kube-api-access-tstdw\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.191913 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.191860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.191913 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.191877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f749f836-83b3-4266-95d7-390a8c19f113-crio-socket\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.191913 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.191901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.201165 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.201138    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:06:50.293157 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.293352 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstdw\" (UniqueName: \"kubernetes.io/projected/f749f836-83b3-4266-95d7-390a8c19f113-kube-api-access-tstdw\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.293352 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.293352 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.293352 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f749f836-83b3-4266-95d7-390a8c19f113-crio-socket\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.293564 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.293564 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.293564 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:50.293463    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.293564 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f749f836-83b3-4266-95d7-390a8c19f113-data-volume\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.293564 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:50.293520    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.793501664 +0000 UTC m=+35.960046232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.293564 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293518    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f749f836-83b3-4266-95d7-390a8c19f113-crio-socket\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.293564 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:50.293537    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.293856 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:50.293598    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.793580681 +0000 UTC m=+35.960125264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.293856 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2t72\" (UniqueName: \"kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.293856 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.293828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f749f836-83b3-4266-95d7-390a8c19f113-data-volume\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.319322 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.319292    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstdw\" (UniqueName: \"kubernetes.io/projected/f749f836-83b3-4266-95d7-390a8c19f113-kube-api-access-tstdw\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.361439 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.361357    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"]
Oct 30 22:06:50.382375 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.382345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"
Oct 30 22:06:50.385921 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.385897    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"]
Oct 30 22:06:50.394180 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.394141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.394806 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.394779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.394909 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.394868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2t72\" (UniqueName: \"kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.394965 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.394908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.395561 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.395536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.396526 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.396502    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.400388 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.400363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.417677 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.417620    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2t72\" (UniqueName: \"kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.477883 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.477845    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:06:50.478070 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.477914    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:50.478070 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.477933    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rrq2z\""
Oct 30 22:06:50.495515 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.495480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plcd4\" (UniqueName: \"kubernetes.io/projected/3c776c71-a37e-48b7-8a6b-e77542ab4bf1-kube-api-access-plcd4\") pod \"migrator-66df5d496b-94228\" (UID: \"3c776c71-a37e-48b7-8a6b-e77542ab4bf1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"
Oct 30 22:06:50.499418 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.499390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:50.604435 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.603152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plcd4\" (UniqueName: \"kubernetes.io/projected/3c776c71-a37e-48b7-8a6b-e77542ab4bf1-kube-api-access-plcd4\") pod \"migrator-66df5d496b-94228\" (UID: \"3c776c71-a37e-48b7-8a6b-e77542ab4bf1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"
Oct 30 22:06:50.606283 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.604679    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-q2bzw"]
Oct 30 22:06:50.627003 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.626936    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plcd4\" (UniqueName: \"kubernetes.io/projected/3c776c71-a37e-48b7-8a6b-e77542ab4bf1-kube-api-access-plcd4\") pod \"migrator-66df5d496b-94228\" (UID: \"3c776c71-a37e-48b7-8a6b-e77542ab4bf1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"
Oct 30 22:06:50.640477 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.640426    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-q2bzw"]
Oct 30 22:06:50.640598 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.640534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.656449 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.656392    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 22:06:50.656580 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.656450    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 22:06:50.656646 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.656597    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 22:06:50.656694 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.656682    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 22:06:50.656734 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.656717    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nfwzj\""
Oct 30 22:06:50.716636 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.716554    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.805661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.805714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.805748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c698726-813c-4953-92df-8cf3917d8bc6-signing-cabundle\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.805779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5wl\" (UniqueName: \"kubernetes.io/projected/8c698726-813c-4953-92df-8cf3917d8bc6-kube-api-access-gz5wl\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.805846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8c698726-813c-4953-92df-8cf3917d8bc6-signing-key\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:50.805990    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:50.806045    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.806028292 +0000 UTC m=+36.972572858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:50.806493    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.806618 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:50.806541    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.806526363 +0000 UTC m=+36.973070930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.823136 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.823104    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cn27f-rccw9"]
Oct 30 22:06:50.827536 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:50.827507    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751bc9b_02bb_4854_959e_0cf6f23cb547.slice/crio-2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473 WatchSource:0}: Error finding container 2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473: Status 404 returned error can't find the container with id 2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473
Oct 30 22:06:50.907109 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.907008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c698726-813c-4953-92df-8cf3917d8bc6-signing-cabundle\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.907109 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.907065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5wl\" (UniqueName: \"kubernetes.io/projected/8c698726-813c-4953-92df-8cf3917d8bc6-kube-api-access-gz5wl\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.907363 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.907133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8c698726-813c-4953-92df-8cf3917d8bc6-signing-key\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.907803 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.907778    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c698726-813c-4953-92df-8cf3917d8bc6-signing-cabundle\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.910387 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.910359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8c698726-813c-4953-92df-8cf3917d8bc6-signing-key\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.928577 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.928551    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5wl\" (UniqueName: \"kubernetes.io/projected/8c698726-813c-4953-92df-8cf3917d8bc6-kube-api-access-gz5wl\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:50.973374 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:50.973341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:51.038815 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:51.038747    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"]
Oct 30 22:06:51.043191 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:51.043151    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c776c71_a37e_48b7_8a6b_e77542ab4bf1.slice/crio-0dafd073e683eb9d3f1aec6d751d71b5e8fb974e5a5707c1f143e09509cdaed2 WatchSource:0}: Error finding container 0dafd073e683eb9d3f1aec6d751d71b5e8fb974e5a5707c1f143e09509cdaed2: Status 404 returned error can't find the container with id 0dafd073e683eb9d3f1aec6d751d71b5e8fb974e5a5707c1f143e09509cdaed2
Oct 30 22:06:51.312167 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:51.312087    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-q2bzw"]
Oct 30 22:06:51.315955 ip-10-0-135-132 kubenswrapper[2576]: W1030 22:06:51.315930    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c698726_813c_4953_92df_8cf3917d8bc6.slice/crio-723f62b3bf1d61d22adf95eff45f41ad4ab4d8852ddb360bacfa6a4235c6546a WatchSource:0}: Error finding container 723f62b3bf1d61d22adf95eff45f41ad4ab4d8852ddb360bacfa6a4235c6546a: Status 404 returned error can't find the container with id 723f62b3bf1d61d22adf95eff45f41ad4ab4d8852ddb360bacfa6a4235c6546a
Oct 30 22:06:51.703640 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:51.703506    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cn27f-rccw9" event={"ID":"7751bc9b-02bb-4854-959e-0cf6f23cb547","Type":"ContainerStarted","Data":"2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473"}
Oct 30 22:06:51.705793 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:51.705761    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw" event={"ID":"8c698726-813c-4953-92df-8cf3917d8bc6","Type":"ContainerStarted","Data":"723f62b3bf1d61d22adf95eff45f41ad4ab4d8852ddb360bacfa6a4235c6546a"}
Oct 30 22:06:51.707153 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:51.707126    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228" event={"ID":"3c776c71-a37e-48b7-8a6b-e77542ab4bf1","Type":"ContainerStarted","Data":"0dafd073e683eb9d3f1aec6d751d71b5e8fb974e5a5707c1f143e09509cdaed2"}
Oct 30 22:06:51.815309 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:51.815272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:51.815487 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:51.815328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:51.815487 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:51.815471    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:51.815586 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:51.815571    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:53.815552081 +0000 UTC m=+38.982096650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:51.815651 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:51.815622    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:51.815701 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:51.815653    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:53.815642427 +0000 UTC m=+38.982186993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:51.915845 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:51.915802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:51.916024 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:51.915950    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:51.916024 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:51.916010    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls podName:ec7d111b-c9ab-455f-acdd-a016d4f48531 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:59.915992463 +0000 UTC m=+45.082537031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls") pod "dns-default-tjbxb" (UID: "ec7d111b-c9ab-455f-acdd-a016d4f48531") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:53.714887 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:53.714843    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228" event={"ID":"3c776c71-a37e-48b7-8a6b-e77542ab4bf1","Type":"ContainerStarted","Data":"3e6064488cee808b7ea1809b35589a851bdf7d7303835dc10e5c5b25a1be95ef"}
Oct 30 22:06:53.755726 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:53.755672    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228" podStartSLOduration=1.9269586539999999 podStartE2EDuration="3.755657386s" totalImagesPullingTime="1.828698732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:53.755074133 +0000 UTC m=+38.921618708" watchObservedRunningTime="2026-10-30 22:06:53.755657386 +0000 UTC m=+38.922201963"
Oct 30 22:06:53.831721 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:53.831684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:53.831881 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:53.831741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:53.831881 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:53.831840    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:53.831976 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:53.831888    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.831873929 +0000 UTC m=+42.998418485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:53.832375 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:53.832266    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:53.832375 ip-10-0-135-132 kubenswrapper[2576]: E1030 22:06:53.832319    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.832305348 +0000 UTC m=+42.998849901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:54.719483 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:54.719442    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw" event={"ID":"8c698726-813c-4953-92df-8cf3917d8bc6","Type":"ContainerStarted","Data":"15854d5a4b066e93a69d0f2709c6541bcb44d9062a0e8e67fe84f6b7b9e83469"}
Oct 30 22:06:54.721738 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:54.721708    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2k9k2" event={"ID":"650a07ef-80ee-44a8-8e96-b6bd104beb37","Type":"ContainerStarted","Data":"a6d85d8c64ba749549619fec535d383c0dd41557458fb0d36f188bdba11f1840"}
Oct 30 22:06:54.722172 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:54.722154    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:54.723939 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:54.723908    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cn27f-rccw9" event={"ID":"7751bc9b-02bb-4854-959e-0cf6f23cb547","Type":"ContainerStarted","Data":"7a2895132c7dc1628bd4190aaf8d40bf0d1d8c740cc5801331ac72ec83da67db"}
Oct 30 22:06:54.761917 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:54.761858    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw" podStartSLOduration=1.486827559 podStartE2EDuration="4.761843188s" totalImagesPullingTime="3.275015629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:54.76016561 +0000 UTC m=+39.926710183" watchObservedRunningTime="2026-10-30 22:06:54.761843188 +0000 UTC m=+39.928387759"
Oct 30 22:06:54.802186 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:54.802122    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2k9k2" podStartSLOduration=35.590592721 podStartE2EDuration="39.802105344s" totalImagesPullingTime="4.211512623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:54.800139078 +0000 UTC m=+39.966683654" watchObservedRunningTime="2026-10-30 22:06:54.802105344 +0000 UTC m=+39.968649919"
Oct 30 22:06:54.841801 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:54.841697    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-cn27f-rccw9" podStartSLOduration=1.5382623610000001 podStartE2EDuration="4.841678622s" totalImagesPullingTime="3.303416261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:54.840416563 +0000 UTC m=+40.006961140" watchObservedRunningTime="2026-10-30 22:06:54.841678622 +0000 UTC m=+40.008223201"
Oct 30 22:06:55.731157 ip-10-0-135-132 kubenswrapper[2576]: I1030 22:06:55.730665    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lcd4g" event={"ID":"099dbe30-a9b3-4166-b289-7fc8e25ff414","Type":"ContainerStarted","Data":"4dd135b8a0792bcf68c640e32558c209cb0b2f8a07bd599cd40b715565cce837"}
Oct 30 22:06:55.886573 ip-10-0-135-132 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 22:06:55.902634 ip-10-0-135-132 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 22:06:55.902813 ip-10-0-135-132 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 22:06:55.902881 ip-10-0-135-132 systemd[1]: kubelet.service: Consumed 2.513s CPU time.
Oct 30 22:06:55.912987 ip-10-0-135-132 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:06:56.002977 ip-10-0-135-132 kubenswrapper[4581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:56.002977 ip-10-0-135-132 kubenswrapper[4581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:06:56.002977 ip-10-0-135-132 kubenswrapper[4581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:56.002977 ip-10-0-135-132 kubenswrapper[4581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:56.005148 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005060    4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.005148 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005148    4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.005148 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005152    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005155    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005158    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005161    4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005163    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005166    4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005169    4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005172    4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005174    4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005184    4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005187    4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005190    4581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005192    4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005195    4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005197    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005200    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005203    4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005205    4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005208    4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.005317 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005210    4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005229    4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005237    4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005244    4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005248    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005250    4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005253    4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005256    4581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005259    4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005261    4581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005264    4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005267    4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005270    4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005274    4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005277    4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005280    4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005283    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005285    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.006075 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005288    4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005305    4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005311    4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005315    4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005320    4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005324    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005329    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005333    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005337    4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005339    4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005342    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005344    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005347    4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005350    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005352    4581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005355    4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005358    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005361    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005363    4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005366    4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.006817 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005369    4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005371    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005374    4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005376    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005379    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005381    4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005384    4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005387    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005389    4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005392    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005394    4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005397    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005399    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005402    4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005404    4581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005408    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005410    4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005413    4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005416    4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005418    4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.007657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005421    4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005423    4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005426    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005428    4581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005431    4581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005433    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005436    4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005439    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005441    4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005444    4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005446    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005449    4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005451    4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005454    4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005456    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005459    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005461    4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005464    4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005466    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005469    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.008462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005471    4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005474    4581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005477    4581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005480    4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005482    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005486    4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.005489    4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006297    4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006304    4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006307    4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006310    4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006313    4581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006315    4581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006321    4581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006324    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006327    4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006330    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006333    4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006335    4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006338    4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.009283 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006341    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006344    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006346    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006349    4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006352    4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006354    4581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006358    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006360    4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006363    4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006365    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006368    4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006371    4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006373    4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006375    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006378    4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006380    4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006383    4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006386    4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006388    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006391    4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.010103 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006394    4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006396    4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006399    4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006401    4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006404    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006407    4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006411    4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006416    4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006420    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006422    4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006425    4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006427    4581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006430    4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006432    4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006435    4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006437    4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006440    4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006442    4581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006445    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.010938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006448    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006450    4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006453    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006455    4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006458    4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006461    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006463    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006466    4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006469    4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006471    4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006474    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006478    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006480    4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006483    4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006486    4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006488    4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006491    4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006494    4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006498    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.011715 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006500    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006502    4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006505    4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006508    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006510    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006513    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006515    4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006518    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006520    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006523    4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006525    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006528    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006531    4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006533    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006536    4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006539    4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006541    4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006544    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006547    4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.012498 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006551    4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006555    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006558    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006560    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006562    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006565    4581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006567    4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006571    4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006574    4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006576    4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006579    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006581    4581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006585    4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006587    4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006589    4581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.006592    4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007084    4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007089    4581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007092    4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007095    4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.013338 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007097    4581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007100    4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007102    4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007105    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007108    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007111    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007114    4581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007116    4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007119    4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007122    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007124    4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007126    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007129    4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007131    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007134    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007136    4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007139    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007141    4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007144    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.014118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007146    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007149    4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007152    4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007154    4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007157    4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007159    4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007171    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007174    4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007177    4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007179    4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007182    4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007185    4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007187    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007190    4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007192    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007195    4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007197    4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007202    4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007205    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007207    4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.014973 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007210    4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007225    4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007228    4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007230    4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007233    4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007236    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007239    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007241    4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007244    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007246    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007250    4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007253    4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007256    4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007259    4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007274    4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007277    4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007280    4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007282    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007285    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.015799 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007288    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007290    4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007293    4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007295    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007298    4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007300    4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007302    4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007306    4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007308    4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007311    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007315    4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007318    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007321    4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007323    4581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007326    4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007328    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007332    4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007334    4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007337    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.016577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007339    4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007342    4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007344    4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007347    4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007349    4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007352    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007354    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007357    4581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007360    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007364    4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007368    4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007371    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007374    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007377    4581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007380    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007383    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007385    4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007388    4581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007391    4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.017512 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007393    4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007396    4581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007398    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007401    4581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007405    4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.007409    4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007508    4581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007516    4581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007522    4581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007526    4581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007531    4581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007534    4581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007540    4581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007546    4581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007549    4581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007552    4581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007556    4581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007559    4581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007562    4581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007565    4581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007572    4581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007576    4581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007579    4581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007589    4581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:06:56.018279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007596    4581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007599    4581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007602    4581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007605    4581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007609    4581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007613    4581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007616    4581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007619    4581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007622    4581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007625    4581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007628    4581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007631    4581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007635    4581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007638    4581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007642    4581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007645    4581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007648    4581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007651    4581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007654    4581 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007657    4581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007665    4581 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007668    4581 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007671    4581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007674    4581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:06:56.019285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007678    4581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007682    4581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007685    4581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007688    4581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007693    4581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007696    4581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007699    4581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007702    4581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007710    4581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007713    4581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007716    4581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007719    4581 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007726    4581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007729    4581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007732    4581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007735    4581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007738    4581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007741    4581 flags.go:64] FLAG: --help="false"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007744    4581 flags.go:64] FLAG: --hostname-override="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007747    4581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007751    4581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007754    4581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007757    4581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:06:56.020327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007761    4581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007764    4581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007767    4581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007770    4581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007803    4581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007807    4581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007810    4581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007813    4581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007816    4581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007818    4581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007821    4581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007824    4581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007827    4581 flags.go:64] FLAG: --lock-file=""
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007830    4581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007834    4581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007837    4581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007843    4581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007846    4581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007855    4581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007858    4581 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007861    4581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007865    4581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007867    4581 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007870    4581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007875    4581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:06:56.021313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007878    4581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007881    4581 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007884    4581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007887    4581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007891    4581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007894    4581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007897    4581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007900    4581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007903    4581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007909    4581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007912    4581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007915    4581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007918    4581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007921    4581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007924    4581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007927    4581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007929    4581 flags.go:64] FLAG: --port="10250"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007932    4581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007935    4581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fd78c64da7cc6732"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007939    4581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007942    4581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007946    4581 flags.go:64] FLAG: --register-node="true"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007948    4581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007952    4581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007955    4581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007958    4581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:06:56.022332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007967    4581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007971    4581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007974    4581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007977    4581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007979    4581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007982    4581 flags.go:64] FLAG: --runonce="false"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007985    4581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007988    4581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007991    4581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007994    4581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.007998    4581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008002    4581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008005    4581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008008    4581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008010    4581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008013    4581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008016    4581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008019    4581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008022    4581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008025    4581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008027    4581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008032    4581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008035    4581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008038    4581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008045    4581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:06:56.023399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008047    4581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008050    4581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008054    4581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008056    4581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008059    4581 flags.go:64] FLAG: --v="2"
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008064    4581 flags.go:64] FLAG: --version="false"
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008068    4581 flags.go:64] FLAG: --vmodule=""
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008072    4581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008075    4581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008245    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008250    4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008253    4581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008255    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008258    4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008261    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008264    4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008266    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008271    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008273    4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008276    4581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008278    4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.024422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008281    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008284    4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008286    4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008289    4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008291    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008294    4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008296    4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008299    4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008301    4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008304    4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008306    4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008309    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008313    4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008316    4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008321    4581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008324    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008327    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008330    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008332    4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.025359 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008334    4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008343    4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008346    4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008349    4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008351    4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008354    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008357    4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008359    4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008362    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008365    4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008368    4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008370    4581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008373    4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008375    4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008378    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008380    4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008383    4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008385    4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008387    4581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.026129 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008390    4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008392    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008394    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008397    4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008399    4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008402    4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008404    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008407    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008411    4581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008413    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008416    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008419    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008421    4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008424    4581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008426    4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008434    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008437    4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008439    4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008442    4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008444    4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.026903 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008447    4581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008449    4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008453    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008455    4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008458    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008461    4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008464    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008466    4581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008469    4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008472    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008474    4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008477    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008479    4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008482    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008484    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008487    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008489    4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008492    4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008494    4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.027717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008496    4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008502    4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008506    4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008509    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008512    4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008514    4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008517    4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008519    4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008521    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008525    4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008527    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008530    4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008532    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008534    4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008537    4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008540    4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.028502 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008542    4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.008569    4581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008606    4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008610    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008613    4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008616    4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008618    4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008621    4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008623    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008626    4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008628    4581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008631    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008634    4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008636    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008639    4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008642    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.029104 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008644    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008649    4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008651    4581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008654    4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008657    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008659    4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008662    4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008664    4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008666    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008672    4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008682    4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008685    4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008688    4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008690    4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008692    4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008695    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008699    4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008703    4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008705    4581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008708    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.031275 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008710    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008713    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008716    4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008718    4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008720    4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008723    4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008726    4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008728    4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008730    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008733    4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008735    4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008738    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008740    4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008744    4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008747    4581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008749    4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008752    4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008754    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008757    4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.032005 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008759    4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008762    4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008764    4581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008767    4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008770    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008772    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008775    4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008777    4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008780    4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008782    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008784    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008787    4581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008789    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008792    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008795    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008797    4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008799    4581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008802    4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008804    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008807    4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.032725 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008809    4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008811    4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008814    4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008816    4581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008819    4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008821    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008825    4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008828    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008831    4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008834    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008836    4581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008839    4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008841    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008844    4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008846    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008848    4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008851    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008855    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008858    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008860    4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.033337 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008862    4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008865    4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008867    4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008871    4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008875    4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008878    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008880    4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008883    4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008886    4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008888    4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008892    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008894    4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.008896    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.012842    4581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.012857    4581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.012877    4581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.012881    4581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012940    4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012946    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.033954 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012950    4581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012952    4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012958    4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012961    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012964    4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012966    4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012969    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012971    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012974    4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012976    4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012979    4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012982    4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012985    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012987    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012990    4581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012993    4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012995    4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.012998    4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013000    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.034549 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013003    4581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013005    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013008    4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013010    4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013013    4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013015    4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013017    4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013020    4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013022    4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013025    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013027    4581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013029    4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013032    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013035    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013037    4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013041    4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013044    4581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013046    4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013049    4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013051    4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.035244 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013054    4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013056    4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013059    4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013061    4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013065    4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013067    4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013070    4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013072    4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013075    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013078    4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013080    4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013082    4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013085    4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013087    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013090    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013092    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013095    4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013097    4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013099    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013102    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.035953 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013104    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013108    4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013111    4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013114    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013117    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013121    4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013125    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013131    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013134    4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013137    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013140    4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013143    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013146    4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013148    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013151    4581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013154    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013156    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013159    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013162    4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.036505 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013164    4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013167    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013169    4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013172    4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013174    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013177    4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013179    4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013182    4581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013184    4581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013187    4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013189    4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013192    4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013195    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013197    4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013199    4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013202    4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013205    4581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013207    4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013210    4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.036974 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013226    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013230    4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013233    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013236    4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013238    4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013242    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013245    4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.013272    4581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013413    4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013418    4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013421    4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013423    4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013426    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013429    4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013432    4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013434    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.037482 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013437    4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013439    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013442    4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013444    4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013447    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013450    4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013452    4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013456    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013459    4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013461    4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013464    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013466    4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013469    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013471    4581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013474    4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013476    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013478    4581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013481    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013485    4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013488    4581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.037882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013490    4581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013494    4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013497    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013500    4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013503    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013506    4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013509    4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013511    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013513    4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013520    4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013523    4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013525    4581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013528    4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013531    4581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013533    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013536    4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013538    4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013541    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013543    4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013546    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.038522 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013549    4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013551    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013554    4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013556    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013559    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013561    4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013564    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013566    4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013568    4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013571    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013575    4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013577    4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013579    4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013582    4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013585    4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013587    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013590    4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013592    4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013595    4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.039257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013597    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013600    4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013602    4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013605    4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013608    4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013610    4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013613    4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013615    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013618    4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013621    4581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013623    4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013626    4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013630    4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013632    4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013634    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013637    4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013639    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013642    4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013644    4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.039789 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013646    4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013649    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013652    4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013654    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013658    4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013660    4581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013663    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013665    4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013668    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013670    4581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013674    4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013678    4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013681    4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013683    4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013686    4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013689    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013692    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013695    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013698    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.040288 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013701    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.013724    4581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013757    4581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013761    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013764    4581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013767    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013771    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013774    4581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013777    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013779    4581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013782    4581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013785    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013787    4581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013789    4581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013792    4581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.040778 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013794    4581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013797    4581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013799    4581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013803    4581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013806    4581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013808    4581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013810    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013813    4581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013816    4581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013818    4581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013821    4581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013823    4581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013826    4581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013828    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013831    4581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013834    4581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013837    4581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013839    4581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013842    4581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013844    4581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.041143 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013846    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013849    4581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013851    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013855    4581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013857    4581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013860    4581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013862    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013865    4581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013867    4581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013869    4581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013872    4581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013874    4581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013877    4581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013879    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013882    4581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013887    4581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013891    4581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013894    4581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013896    4581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.041659 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013899    4581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013902    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013904    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013907    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013910    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013912    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013915    4581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013917    4581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013920    4581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013922    4581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013925    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013927    4581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013930    4581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013932    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013935    4581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013937    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013941    4581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013944    4581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013946    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.042201 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013949    4581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013951    4581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013953    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013956    4581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013958    4581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013960    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013963    4581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013965    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013967    4581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013971    4581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013973    4581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013976    4581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013978    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013981    4581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013983    4581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013986    4581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013989    4581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013993    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013995    4581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.013998    4581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.042719 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014000    4581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014003    4581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014005    4581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014007    4581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014010    4581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014013    4581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014015    4581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014019    4581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014022    4581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014025    4581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014028    4581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014030    4581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014032    4581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014035    4581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:56.014037    4581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.014088    4581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.015042    4581 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.015090    4581 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.015296    4581 server.go:1014] "Starting client certificate rotation"
Oct 30 22:06:56.043232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.015407    4581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:56.043690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.015464    4581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:01:15 +0000 UTC" deadline="2028-04-24 21:42:35.616337371 +0000 UTC"
Oct 30 22:06:56.043690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.015513    4581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13007h35m39.600827385s"
Oct 30 22:06:56.043690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.015762    4581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:56.043690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.031366    4581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:56.043690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.032969    4581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:06:56.043690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.034698    4581 log.go:25] "Validated CRI v1 image API"
Oct 30 22:06:56.043690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.035506    4581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:06:56.043690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.037104    4581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8325bcd5-ce36-42cd-b331-bc78295d5e01:/dev/nvme0n1p4 90fab635-ac8b-41e6-9721-b7562f79524b:/dev/nvme0n1p3]
Oct 30 22:06:56.043908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.037121    4581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04d9f44dea51a8df5db2e7029036128e537ed479d898f6d84aee7c0610bff3fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04d9f44dea51a8df5db2e7029036128e537ed479d898f6d84aee7c0610bff3fa/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dafd073e683eb9d3f1aec6d751d71b5e8fb974e5a5707c1f143e09509cdaed2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dafd073e683eb9d3f1aec6d751d71b5e8fb974e5a5707c1f143e09509cdaed2/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2abf750bbc0db41984c5c318d945be8bc50daabe03cace872f1c43ba72be5a44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2abf750bbc0db41984c5c318d945be8bc50daabe03cace872f1c43ba72be5a44/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3978efc022e99c0ed1477c13e15b2665f5ba7b5ff9f9bfb9f6ec7d231e74c88f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3978efc022e99c0ed1477c13e15b2665f5ba7b5ff9f9bfb9f6ec7d231e74c88f/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ef9ef4c440384a67fd36c991c0814e7c1b29a5ff4761cda9e2f51a171c0b7b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ef9ef4c440384a67fd36c991c0814e7c1b29a5ff4761cda9e2f51a171c0b7b7/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fbd3473f977c340d1879e65b736833ea0ae5b94dfb876b9032aab258f9a0484/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fbd3473f977c340d1879e65b736833ea0ae5b94dfb876b9032aab258f9a0484/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47bc1a3636731a6bc141ab3e482018f455c62c071f4f4d566f2ad452c4dfe579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47bc1a3636731a6bc141ab3e482018f455c62c071f4f4d566f2ad452c4dfe579/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eb7586174bc6b86c522d8f2b9e84000ec80a3849b6f217acf7459ce2fd9980d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eb7586174bc6b86c522d8f2b9e84000ec80a3849b6f217acf7459ce2fd9980d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59adf563b1f1cc53ffcd5dcaae43b926565cad753f1bb334086797eed8a8c65c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59adf563b1f1cc53ffcd5dcaae43b926565cad753f1bb334086797eed8a8c65c/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5adcb1750e290d2acefa7779ffc18568ab27e26eb820d97a29b358ab3927c61b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5adcb1750e290d2acefa7779ffc18568ab27e26eb820d97a29b358ab3927c61b/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/723f62b3bf1d61d22adf95eff45f41ad4ab4d8852ddb360bacfa6a4235c6546a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/723f62b3bf1d61d22adf95eff45f41ad4ab4d8852ddb360bacfa6a4235c6546a/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a75252d36590d5bfc165a9678835be2ac5f2a31183b5b1962b1ec4fcaaff560/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a75252d36590d5bfc165a9678835be2ac5f2a31183b5b1962b1ec4fcaaff560/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a2a70f900401dfa0c683d458fb33c586efa7eda71e387ef35f684a1d75a0074/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a2a70f900401dfa0c683d458fb33c586efa7eda71e387ef35f684a1d75a0074/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af13edebc8f85b896b61c7a475e84342ca69c450d793816b00bebbe13e0a9b35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af13edebc8f85b896b61c7a475e84342ca69c450d793816b00bebbe13e0a9b35/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b744483fcd9cd00c46423a3d59c46e07919a6ff9e8e38a37226dec01ee572ab6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b744483fcd9cd00c46423a3d59c46e07919a6ff9e8e38a37226dec01ee572ab6/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/099dbe30-a9b3-4166-b289-7fc8e25ff414/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/099dbe30-a9b3-4166-b289-7fc8e25ff414/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10654728-36ab-4482-9fcb-775d88ab5aa0/volumes/kubernetes.io~projected/kube-api-access-bgr5b:{mountpoint:/var/lib/kubelet/pods/10654728-36ab-4482-9fcb-775d88ab5aa0/volumes/kubernetes.io~projected/kube-api-access-bgr5b major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16934604-e78f-4968-bdcd-3e48bd4f6882/volumes/kubernetes.io~projected/kube-api-access-5gbgm:{mountpoint:/var/lib/kubelet/pods/16934604-e78f-4968-bdcd-3e48bd4f6882/volumes/kubernetes.io~projected/kube-api-access-5gbgm major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16934604-e78f-4968-bdcd-3e48bd4f6882/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/16934604-e78f-4968-bdcd-3e48bd4f6882/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c776c71-a37e-48b7-8a6b-e77542ab4bf1/volumes/kubernetes.io~projected/kube-api-access-plcd4:{mountpoint:/var/lib/kubelet/pods/3c776c71-a37e-48b7-8a6b-e77542ab4bf1/volumes/kubernetes.io~projected/kube-api-access-plcd4 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/543555d6-bbb3-4643-8198-9c7c397c07d9/volumes/kubernetes.io~projected/kube-api-access-tdd6d:{mountpoint:/var/lib/kubelet/pods/543555d6-bbb3-4643-8198-9c7c397c07d9/volumes/kubernetes.io~projected/kube-api-access-tdd6d major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/650a07ef-80ee-44a8-8e96-b6bd104beb37/volumes/kubernetes.io~projected/kube-api-access-r4r4g:{mountpoint:/var/lib/kubelet/pods/650a07ef-80ee-44a8-8e96-b6bd104beb37/volumes/kubernetes.io~projected/kube-api-access-r4r4g major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7751bc9b-02bb-4854-959e-0cf6f23cb547/volumes/kubernetes.io~projected/kube-api-access-b2t72:{mountpoint:/var/lib/kubelet/pods/7751bc9b-02bb-4854-959e-0cf6f23cb547/volumes/kubernetes.io~projected/kube-api-access-b2t72 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~projected/kube-api-access-mvqnf:{mountpoint:/var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~projected/kube-api-access-mvqnf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b91ef29-ba61-4265-a756-ac5ad6ab604d/volumes/kubernetes.io~projected/kube-api-access-ckxhf:{mountpoint:/var/lib/kubelet/pods/8b91ef29-ba61-4265-a756-ac5ad6ab604d/volumes/kubernetes.io~projected/kube-api-access-ckxhf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c698726-813c-4953-92df-8cf3917d8bc6/volumes/kubernetes.io~projected/kube-api-access-gz5wl:{mountpoint:/var/lib/kubelet/pods/8c698726-813c-4953-92df-8cf3917d8bc6/volumes/kubernetes.io~projected/kube-api-access-gz5wl major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c698726-813c-4953-92df-8cf3917d8bc6/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/8c698726-813c-4953-92df-8cf3917d8bc6/volumes/kubernetes.io~secret/signing-key major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91c86d61-12bf-4446-b98a-f9e571ad5ffb/volumes/kubernetes.io~projected/kube-api-access-qc4fx:{mountpoint:/var/lib/kubelet/pods/91c86d61-12bf-4446-b98a-f9e571ad5ffb/volumes/kubernetes.io~projected/kube-api-access-qc4fx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99488cc5-1b6c-4b64-a38e-6403a080f2a1/volumes/kubernetes.io~projected/kube-api-access-rqnrc:{mountpoint:/var/lib/kubelet/pods/99488cc5-1b6c-4b64-a38e-6403a080f2a1/volumes/kubernetes.io~projected/kube-api-access-rqnrc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volumes/kubernetes.io~projected/kube-api-access-k8fbd:{mountpoint:/var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volumes/kubernetes.io~projected/kube-api-access-k8fbd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7665c55-0b43-4318-9bad-342fce7a359d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c7665c55-0b43-4318-9bad-342fce7a359d/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dece32b6-cb70-479d-9a12-bc2608b23129/volumes/kubernetes.io~projected/kube-api-access-85f74:{mountpoint:/var/lib/kubelet/pods/dece32b6-cb70-479d-9a12-bc2608b23129/volumes/kubernetes.io~projected/kube-api-access-85f74 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df124a61-d687-4ef9-a732-2d5def6ef79c/volumes/kubernetes.io~projected/kube-api-access-fnpjh:{mountpoint:/var/lib/kubelet/pods/df124a61-d687-4ef9-a732-2d5def6ef79c/volumes/kubernetes.io~projected/kube-api-access-fnpjh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec7d111b-c9ab-455f-acdd-a016d4f48531/volumes/kubernetes.io~projected/kube-api-access-bdr8r:{mountpoint:/var/lib/kubelet/pods/ec7d111b-c9ab-455f-acdd-a016d4f48531/volumes/kubernetes.io~projected/kube-api-access-bdr8r major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f749f836-83b3-4266-95d7-390a8c19f113/volumes/kubernetes.io~projected/kube-api-access-tstdw:{mountpoint:/var/lib/kubelet/pods/f749f836-83b3-4266-95d7-390a8c19f113/volumes/kubernetes.io~projected/kube-api-access-tstdw major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/129fb2be7cddd57fe238afb74625f2558d3f002988042c790b5487b6cdc7d98d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/771f6716d615212c2cfcd05e3bd38dc0051cca652e67cccf6c70441559fe06e1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/020a2ace48caaf4b94a09fa7a95ef2a3846b8aec2a5477755eb2def5150a743a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d4054167e55fc726349869c022bc23051c8cbac5ebc54e1be272a88024a58da6/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bba5072a392fe3084243d31868642e5377cb0fddf7e7be5325ea8b8c84073dfc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8f6c4e8ef13b8aa2e2c805dab51a501600ce68b81aef07d7715846ae76fbdf32/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a20aade4a756208684574432c7b62c29d673d32fcba4ae4c9678091d2bd4795e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/685664d4ea3569025a648ba10623523a066480a5cf25e23068ccd5513392eacf/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f459ec175f941454d994439060a7ae39f983432e510775ef66ea30807085ccd1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/39e3000414a757f62a48c530e19a1cbc830f8e12bb2bf45b3771dbee35f598ed/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/75f7567b1f5145b0745b1df60be342ba441fe8d5617033e197ca34feb39d4e98/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/7cefff0c9ee21006377f897412b5d4ec3be431caa870c3aa63950a9510573f93/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/166314ac9d9b8c30668f7c0d0f7ed8642b6542dbd91e8db72d85b91d8636ab68/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3838994daec1d4a11b877009635df55436f4050e08b90b163645078d485e19cd/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cdaaa8890c6da8b7dcf05e648ed9f6284c71e70ab33282fbc7985e6824e8455b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/695f7f579a17508c40427f547eba6f1bf0e77a32c9c159a00dfa858aa046ef1d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/22c80072055aab61b7d1e96e93640bf5c52da16f9bafa7db5212a79f068ccf49/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0068ad6350354aa1d75f2a40441caf6026bc324df17a8bd00ab1c2c1e1d233db/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e755c1a2ee74ebdbfc4ff8d7e829bbcaff4c7c37bfd52d3fba00867883678080/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/41089cc64577675267275eb64a708c5c4171f07010a23bfb0e3c6d081263e410/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e13b9d8b51479be2502a206af9e424789ba63295360358c89d9de12c8e77f18a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/9f0efe8674a2e80bdd712dec58c2cac4f74779ae6b02ea8e838683a5f75517d8/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/956c41ca9acc7e5967f816680b1d49c552d53885bb32c6152e4624454c4ffeb7/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/aa7f43173ffca9f3f08baf82a341efefa5f4b5c115663f718c4512923cddfe3f/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/7b26797a42db42891ef3c65b618d1f3233b810888025687e3827fe5e989df37e/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/05ed8dd4df51b49be06d26286a39bc06a39a384e4d1033bc413170e0c52f7f1d/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/d7382715d475ccb21f584f40f6a9c7b285c90d769505238bde176e12e54423b1/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/6d02ed45bf5b647447e541922b32f1071a02482992efb61a6709a5810bf98507/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/9556f48ff55b7b487409a8c7c57f8a726efa4dbd7a6af7949e69f7fe278944d1/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/3f05f3899f83676365b16d3fd42c6b5eec81281fedcf51f19c0499088e964ce6/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/84826096262a7a45d860b7271fed1514aae4db33b0dcdd0ca87fef4bc70e016a/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/34b13ead95be7f61a9cd731c4db1312ad18172ff7b8c2d9ae91900f46d80dea9/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/ad6989f603013a872321a03dada36c8159291cc30a2aa9b4bd52e7b4654a0f49/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/57445fd4ea59ca69bbe8cb41b8fc2931a8708133f5fbbb77061e6524cec91467/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/504943c96a8c064f689b8a54bd5d3db9bab9e15d6c836705ac6fdf8246c14b69/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/39821d32418d1320762c53c07f4a615bf751ab36799356a2900252bc1d8c36f2/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/f35dc203ec49d3123dfb1065de5ae08bbff24de210f8258b245330181cf60d6c/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/bdfb6d5d02294cb154998085073bd84f4ac256eef4591678b83714b74e4e5f4e/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a8fd2a7781c4823f930f728d90b39c36f595f4c9e8fa57bfb16c3ed5c92361bf/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/fc0edd6a7f92478b0880ddbf2171af622ce0432314d2a30adb7ca93de27350c3/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/bd3ecd22e7d57132901e6c98a42cde3149aa880237095d998d6e6d117ded1797/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/39fe9a0d8e29f38f2c67e78dcce80b0042332b4a4138daf035678974834dd0b4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/d09c3ff1bf90eae6cf22a4052249ee5511957daf17f65e4f3d121edd38062f63/merged major:0 minor:97 fsType:overlay blockSize:0}]
Oct 30 22:06:56.044120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.043861    4581 manager.go:217] Machine: {Timestamp:2026-10-30 22:06:56.043405154 +0000 UTC m=+0.082344032 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099008 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e6b413026cb8178cded090f9d37c8 SystemUUID:ec2e6b41-3026-cb81-78cd-ed090f9d37c8 BootID:c59ef6ca-1df8-4ea4-a5e5-0a96b04b60db Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/91c86d61-12bf-4446-b98a-f9e571ad5ffb/volumes/kubernetes.io~projected/kube-api-access-qc4fx DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eb7586174bc6b86c522d8f2b9e84000ec80a3849b6f217acf7459ce2fd9980d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5adcb1750e290d2acefa7779ffc18568ab27e26eb820d97a29b358ab3927c61b/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/723f62b3bf1d61d22adf95eff45f41ad4ab4d8852ddb360bacfa6a4235c6546a/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04d9f44dea51a8df5db2e7029036128e537ed479d898f6d84aee7c0610bff3fa/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b744483fcd9cd00c46423a3d59c46e07919a6ff9e8e38a37226dec01ee572ab6/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/099dbe30-a9b3-4166-b289-7fc8e25ff414/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3978efc022e99c0ed1477c13e15b2665f5ba7b5ff9f9bfb9f6ec7d231e74c88f/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99488cc5-1b6c-4b64-a38e-6403a080f2a1/volumes/kubernetes.io~projected/kube-api-access-rqnrc DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a2a70f900401dfa0c683d458fb33c586efa7eda71e387ef35f684a1d75a0074/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f749f836-83b3-4266-95d7-390a8c19f113/volumes/kubernetes.io~projected/kube-api-access-tstdw DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ef9ef4c440384a67fd36c991c0814e7c1b29a5ff4761cda9e2f51a171c0b7b7/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2abf750bbc0db41984c5c318d945be8bc50daabe03cace872f1c43ba72be5a44/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47bc1a3636731a6bc141ab3e482018f455c62c071f4f4d566f2ad452c4dfe579/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dafd073e683eb9d3f1aec6d751d71b5e8fb974e5a5707c1f143e09509cdaed2/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dece32b6-cb70-479d-9a12-bc2608b23129/volumes/kubernetes.io~projected/kube-api-access-85f74 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c776c71-a37e-48b7-8a6b-e77542ab4bf1/volumes/kubernetes.io~projected/kube-api-access-plcd4 DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16934604-e78f-4968-bdcd-3e48bd4f6882/volumes/kubernetes.io~projected/kube-api-access-5gbgm DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/df124a61-d687-4ef9-a732-2d5def6ef79c/volumes/kubernetes.io~projected/kube-api-access-fnpjh DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f07db13-af88-4f66-a3ea-23b4314d6144/volumes/kubernetes.io~projected/kube-api-access-mvqnf DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59adf563b1f1cc53ffcd5dcaae43b926565cad753f1bb334086797eed8a8c65c/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fbd3473f977c340d1879e65b736833ea0ae5b94dfb876b9032aab258f9a0484/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10654728-36ab-4482-9fcb-775d88ab5aa0/volumes/kubernetes.io~projected/kube-api-access-bgr5b DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a75252d36590d5bfc165a9678835be2ac5f2a31183b5b1962b1ec4fcaaff560/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c698726-813c-4953-92df-8cf3917d8bc6/volumes/kubernetes.io~projected/kube-api-access-gz5wl DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/543555d6-bbb3-4643-8198-9c7c397c07d9/volumes/kubernetes.io~projected/kube-api-access-tdd6d DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/650a07ef-80ee-44a8-8e96-b6bd104beb37/volumes/kubernetes.io~projected/kube-api-access-r4r4g DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16934604-e78f-4968-bdcd-3e48bd4f6882/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7751bc9b-02bb-4854-959e-0cf6f23cb547/volumes/kubernetes.io~projected/kube-api-access-b2t72 DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7665c55-0b43-4318-9bad-342fce7a359d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b91ef29-ba61-4265-a756-ac5ad6ab604d/volumes/kubernetes.io~projected/kube-api-access-ckxhf DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af13edebc8f85b896b61c7a475e84342ca69c450d793816b00bebbe13e0a9b35/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c698726-813c-4953-92df-8cf3917d8bc6/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad9995b8-ba05-47ba-81c6-6f74c234f2c1/volumes/kubernetes.io~projected/kube-api-access-k8fbd DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec7d111b-c9ab-455f-acdd-a016d4f48531/volumes/kubernetes.io~projected/kube-api-access-bdr8r DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0dafd073e683eb9 MacAddress:2e:74:bc:ad:52:4c Speed:10000 Mtu:8901} {Name:2d64a251ffb120d MacAddress:be:31:e8:97:22:13 Speed:10000 Mtu:8901} {Name:3fbd3473f977c34 MacAddress:d2:ec:74:ec:ad:a4 Speed:10000 Mtu:8901} {Name:47bc1a3636731a6 MacAddress:e2:66:46:fa:ce:71 Speed:10000 Mtu:8901} {Name:5adcb1750e290d2 MacAddress:26:77:19:24:2f:16 Speed:10000 Mtu:8901} {Name:723f62b3bf1d61d MacAddress:a2:d7:5f:03:c4:05 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c3:d3:f5:eb Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:bb:57:57:da:b8 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:c3:d3:f5:eb Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:12:2c:12:85:ac:ee Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:df:79:89:c5:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:06:56.044291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044124    4581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:06:56.044291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044166    4581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:06:56.044480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044456    4581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:06:56.044609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044482    4581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:06:56.044655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044612    4581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:06:56.044655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044620    4581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:06:56.044655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044633    4581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:56.044655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044652    4581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:56.044754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044739    4581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:06:56.044897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044887    4581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:06:56.045001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.044992    4581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:06:56.045043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.045027    4581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:06:56.045043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.045038    4581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:06:56.045109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.045047    4581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:06:56.045109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.045057    4581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:06:56.045109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.045067    4581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:06:56.045679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.045663    4581 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:06:56.045679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.045678    4581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:56.046784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.046763    4581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:06:56.049719 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.049700    4581 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:06:56.049719 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.049717    4581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:56.049871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.049852    4581 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 22:06:56.050081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050066    4581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:06:56.050144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050106    4581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:06:56.050245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050234    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:06:56.050298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050253    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:06:56.050298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050263    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:06:56.050298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050272    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:06:56.050298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050281    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:06:56.050298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050291    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:06:56.050525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050300    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:06:56.050525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050309    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:06:56.050525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050321    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:06:56.050525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050330    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:06:56.050525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050345    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:06:56.050525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050360    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:06:56.050525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050397    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:06:56.050525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.050407    4581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:06:56.053892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.053875    4581 server.go:1263] "Started kubelet"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.054094    4581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.054159    4581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.054244    4581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.054550    4581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.056053    4581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.056945    4581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.057033    4581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.057105    4581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:17 +0000 UTC" deadline="2028-04-24 23:29:44.339845701 +0000 UTC"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.057141    4581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13009h22m48.282707816s"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:56.057185    4581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-132.ec2.internal\" not found"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.057248    4581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.057257    4581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:06:56.058322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.057401    4581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:06:56.054456 ip-10-0-135-132 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059735    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f07db13-af88-4f66-a3ea-23b4314d6144" volumeName="kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-tmp" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059771    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f07db13-af88-4f66-a3ea-23b4314d6144" volumeName="kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-tuned" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059788    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b91ef29-ba61-4265-a756-ac5ad6ab604d" volumeName="kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059802    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b91ef29-ba61-4265-a756-ac5ad6ab604d" volumeName="kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059816    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99488cc5-1b6c-4b64-a38e-6403a080f2a1" volumeName="kubernetes.io/projected/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kube-api-access-rqnrc" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059828    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dece32b6-cb70-479d-9a12-bc2608b23129" volumeName="kubernetes.io/projected/dece32b6-cb70-479d-9a12-bc2608b23129-kube-api-access-85f74" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059842    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7751bc9b-02bb-4854-959e-0cf6f23cb547" volumeName="kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059854    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b91ef29-ba61-4265-a756-ac5ad6ab604d" volumeName="kubernetes.io/projected/8b91ef29-ba61-4265-a756-ac5ad6ab604d-kube-api-access-ckxhf" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059869    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c698726-813c-4953-92df-8cf3917d8bc6" volumeName="kubernetes.io/configmap/8c698726-813c-4953-92df-8cf3917d8bc6-signing-cabundle" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059884    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9995b8-ba05-47ba-81c6-6f74c234f2c1" volumeName="kubernetes.io/projected/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-kube-api-access-k8fbd" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059896    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7665c55-0b43-4318-9bad-342fce7a359d" volumeName="kubernetes.io/configmap/c7665c55-0b43-4318-9bad-342fce7a359d-konnectivity-ca" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059908    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dece32b6-cb70-479d-9a12-bc2608b23129" volumeName="kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059920    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec7d111b-c9ab-455f-acdd-a016d4f48531" volumeName="kubernetes.io/projected/ec7d111b-c9ab-455f-acdd-a016d4f48531-kube-api-access-bdr8r" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059932    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f749f836-83b3-4266-95d7-390a8c19f113" volumeName="kubernetes.io/empty-dir/f749f836-83b3-4266-95d7-390a8c19f113-data-volume" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059948    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7751bc9b-02bb-4854-959e-0cf6f23cb547" volumeName="kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059960    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9995b8-ba05-47ba-81c6-6f74c234f2c1" volumeName="kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-env-overrides" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059972    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9995b8-ba05-47ba-81c6-6f74c234f2c1" volumeName="kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059986    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f749f836-83b3-4266-95d7-390a8c19f113" volumeName="kubernetes.io/projected/f749f836-83b3-4266-95d7-390a8c19f113-kube-api-access-tstdw" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.059998    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10654728-36ab-4482-9fcb-775d88ab5aa0" volumeName="kubernetes.io/projected/10654728-36ab-4482-9fcb-775d88ab5aa0-kube-api-access-bgr5b" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060010    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f07db13-af88-4f66-a3ea-23b4314d6144" volumeName="kubernetes.io/projected/7f07db13-af88-4f66-a3ea-23b4314d6144-kube-api-access-mvqnf" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060022    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c698726-813c-4953-92df-8cf3917d8bc6" volumeName="kubernetes.io/projected/8c698726-813c-4953-92df-8cf3917d8bc6-kube-api-access-gz5wl" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060036    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91c86d61-12bf-4446-b98a-f9e571ad5ffb" volumeName="kubernetes.io/configmap/91c86d61-12bf-4446-b98a-f9e571ad5ffb-serviceca" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060049    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9995b8-ba05-47ba-81c6-6f74c234f2c1" volumeName="kubernetes.io/secret/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060060    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7665c55-0b43-4318-9bad-342fce7a359d" volumeName="kubernetes.io/secret/c7665c55-0b43-4318-9bad-342fce7a359d-agent-certs" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060073    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec7d111b-c9ab-455f-acdd-a016d4f48531" volumeName="kubernetes.io/configmap/ec7d111b-c9ab-455f-acdd-a016d4f48531-config-volume" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060086    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16934604-e78f-4968-bdcd-3e48bd4f6882" volumeName="kubernetes.io/projected/16934604-e78f-4968-bdcd-3e48bd4f6882-kube-api-access-5gbgm" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060099    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7751bc9b-02bb-4854-959e-0cf6f23cb547" volumeName="kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060112    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df124a61-d687-4ef9-a732-2d5def6ef79c" volumeName="kubernetes.io/projected/df124a61-d687-4ef9-a732-2d5def6ef79c-kube-api-access-fnpjh" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060127    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10654728-36ab-4482-9fcb-775d88ab5aa0" volumeName="kubernetes.io/configmap/10654728-36ab-4482-9fcb-775d88ab5aa0-iptables-alerter-script" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060145    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16934604-e78f-4968-bdcd-3e48bd4f6882" volumeName="kubernetes.io/secret/16934604-e78f-4968-bdcd-3e48bd4f6882-cert" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060159    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c776c71-a37e-48b7-8a6b-e77542ab4bf1" volumeName="kubernetes.io/projected/3c776c71-a37e-48b7-8a6b-e77542ab4bf1-kube-api-access-plcd4" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060171    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b91ef29-ba61-4265-a756-ac5ad6ab604d" volumeName="kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060183    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c698726-813c-4953-92df-8cf3917d8bc6" volumeName="kubernetes.io/secret/8c698726-813c-4953-92df-8cf3917d8bc6-signing-key" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060198    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91c86d61-12bf-4446-b98a-f9e571ad5ffb" volumeName="kubernetes.io/projected/91c86d61-12bf-4446-b98a-f9e571ad5ffb-kube-api-access-qc4fx" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060210    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9995b8-ba05-47ba-81c6-6f74c234f2c1" volumeName="kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-config" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060240    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dece32b6-cb70-479d-9a12-bc2608b23129" volumeName="kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-multus-daemon-config" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060252    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="099dbe30-a9b3-4166-b289-7fc8e25ff414" volumeName="kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060274    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="543555d6-bbb3-4643-8198-9c7c397c07d9" volumeName="kubernetes.io/projected/543555d6-bbb3-4643-8198-9c7c397c07d9-kube-api-access-tdd6d" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060286    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="650a07ef-80ee-44a8-8e96-b6bd104beb37" volumeName="kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060300    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df124a61-d687-4ef9-a732-2d5def6ef79c" volumeName="kubernetes.io/empty-dir/df124a61-d687-4ef9-a732-2d5def6ef79c-tmp-dir" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060313    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec7d111b-c9ab-455f-acdd-a016d4f48531" volumeName="kubernetes.io/empty-dir/ec7d111b-c9ab-455f-acdd-a016d4f48531-tmp-dir" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060325    4581 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7751bc9b-02bb-4854-959e-0cf6f23cb547" volumeName="kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle" seLinuxMountContext=""
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060336    4581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:06:56.060705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.060344    4581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:06:56.070597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.066746    4581 factory.go:55] Registering systemd factory
Oct 30 22:06:56.070597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.066779    4581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:06:56.070597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.067658    4581 factory.go:153] Registering CRI-O factory
Oct 30 22:06:56.070597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.067674    4581 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:06:56.070597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.067729    4581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:06:56.070597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.067748    4581 factory.go:103] Registering Raw factory
Oct 30 22:06:56.070597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.067769    4581 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:06:56.070597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.069892    4581 manager.go:319] Starting recovery of all containers
Oct 30 22:06:56.085277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.085210    4581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:06:56.087807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.087763    4581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:06:56.087807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.087789    4581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:06:56.087807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.087807    4581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:06:56.089471 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:56.087839    4581 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 22:06:56.093646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.093624    4581 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 22:06:56.094190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.094019    4581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:56.097793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.097493    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:06:56.098332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.097911    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:06:56.099417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.099261    4581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:56.108414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.108381    4581 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="6cd2fc344130ad7ef77f228d73a274255e86016f6eb4ba459ea3a3f088e6722e" exitCode=0
Oct 30 22:06:56.108414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.108416    4581 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="e3ed1ed24427a46ffb3685bb8ff495e6344df7aa6286e3169bbad37782937c80" exitCode=0
Oct 30 22:06:56.108806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.108424    4581 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="96726f20f4c01f736c3ada0d7e11311434081ad6143c963e1478483b990698ee" exitCode=0
Oct 30 22:06:56.108806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.108431    4581 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="5faaf6c7d3ccacd9e477dc0638ed89cac71e992c47e6136508b19bb8c7639b5d" exitCode=0
Oct 30 22:06:56.108806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.108439    4581 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="d2c26e55e5cd8bc1f1910376c0c0f13b13d8ccc6b3809b9d0ec7b99c4f102f6d" exitCode=0
Oct 30 22:06:56.108806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.108444    4581 generic.go:356] "Generic (PLEG): container finished" podID="8b91ef29-ba61-4265-a756-ac5ad6ab604d" containerID="bbf8617d8500207a12c2243b9df4b3ba11f0fc3b2b3021739a00d55bc92ebc33" exitCode=0
Oct 30 22:06:56.109041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.109026    4581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:56.112728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.112711    4581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:56.116583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.115929    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:06:56.118037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.116806    4581 generic.go:356] "Generic (PLEG): container finished" podID="ad9995b8-ba05-47ba-81c6-6f74c234f2c1" containerID="e0aff1a9d7e122e6e880aba12f8d7e9e920822bae4e96f27d08978f8b911071b" exitCode=1
Oct 30 22:06:56.127831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.127807    4581 generic.go:356] "Generic (PLEG): container finished" podID="d7605b2d536800c86463888458ce09c5" containerID="a883995867f6b4dab01af83bc8d2b5a9e960df40397dcb2b97b44cfb7c297f7d" exitCode=0
Oct 30 22:06:56.145035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.144973    4581 manager.go:324] Recovery completed
Oct 30 22:06:56.165003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.164980    4581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:06:56.165003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.164996    4581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:06:56.165188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165013    4581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:06:56.165188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165148    4581 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 22:06:56.165188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165162    4581 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 22:06:56.165188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165186    4581 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 22:06:56.165342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165195    4581 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 22:06:56.165342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165202    4581 policy_none.go:50] "Start"
Oct 30 22:06:56.165342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165231    4581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:06:56.165342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165244    4581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:06:56.165469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165428    4581 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 22:06:56.165469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165437    4581 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 22:06:56.165469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.165444    4581 policy_none.go:44] "Start"
Oct 30 22:06:56.169555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.169524    4581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:06:56.172637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.172623    4581 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:06:56.173256 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:56.172651    4581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:06:56.173345 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.173259    4581 server.go:87] "Starting device plugin registration server"
Oct 30 22:06:56.173559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.173543    4581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:06:56.173630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.173558    4581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:06:56.173683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.173644    4581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:06:56.173786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.173747    4581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:06:56.173888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.173794    4581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:06:56.173888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.173800    4581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:06:56.173991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.173850    4581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:06:56.173783485Z","UUID":"6e43cccb-39f0-41d7-8a03-b5e7e34c0159","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:06:56.175531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.175508    4581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:06:56.175634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.175550    4581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:06:56.175763 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:56.175742    4581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:06:56.188346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.188310    4581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"]
Oct 30 22:06:56.189240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.189198    4581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.189489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.189433    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal" event={"ID":"3f5710f91ddba841cbff091b973dc1da","Type":"ContainerStarted","Data":"baf484fd55e32ecbabd2eedfdf411b21854c741975e73ed62ff6b41d5fe4ae77"}
Oct 30 22:06:56.189554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.189510    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal" event={"ID":"3f5710f91ddba841cbff091b973dc1da","Type":"ContainerStarted","Data":"4eb7586174bc6b86c522d8f2b9e84000ec80a3849b6f217acf7459ce2fd9980d"}
Oct 30 22:06:56.189554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.189528    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal" event={"ID":"d7605b2d536800c86463888458ce09c5","Type":"ContainerStarted","Data":"261b87bc22d1c812cf3583657b33c99fdf5931d36d10a6ec009fc7e58d6a7125"}
Oct 30 22:06:56.189554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.189548    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal" event={"ID":"d7605b2d536800c86463888458ce09c5","Type":"ContainerDied","Data":"a883995867f6b4dab01af83bc8d2b5a9e960df40397dcb2b97b44cfb7c297f7d"}
Oct 30 22:06:56.189695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.189569    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal" event={"ID":"d7605b2d536800c86463888458ce09c5","Type":"ContainerStarted","Data":"04d9f44dea51a8df5db2e7029036128e537ed479d898f6d84aee7c0610bff3fa"}
Oct 30 22:06:56.189797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.189782    4581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.209566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.209532    4581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:56.209685 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:56.209585    4581 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-132.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.210171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.210155    4581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:56.210256 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:56.210240    4581 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.274184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.274149    4581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:56.275583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.275566    4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:56.275688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.275596    4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:56.275688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.275607    4581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:56.275688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.275639    4581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.294058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.294024    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f5710f91ddba841cbff091b973dc1da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-132.ec2.internal\" (UID: \"3f5710f91ddba841cbff091b973dc1da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.294186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.294112    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.294186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.294139    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.307690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.307664    4581 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.307802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.307732    4581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.395261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.395151    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.395261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.395195    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.395261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.395237    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f5710f91ddba841cbff091b973dc1da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-132.ec2.internal\" (UID: \"3f5710f91ddba841cbff091b973dc1da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.395523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.395293    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f5710f91ddba841cbff091b973dc1da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-132.ec2.internal\" (UID: \"3f5710f91ddba841cbff091b973dc1da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.395523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.395295    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:56.395523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:56.395305    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7605b2d536800c86463888458ce09c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" (UID: \"d7605b2d536800c86463888458ce09c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:57.046202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.046167    4581 apiserver.go:52] "Watching apiserver"
Oct 30 22:06:57.088812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.088783    4581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:06:57.089298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.089278    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-l5shw","openshift-network-diagnostics/network-check-target-2k9k2","openshift-network-operator/iptables-alerter-8kmq8","openshift-ingress-canary/ingress-canary-456zn","openshift-insights/insights-runtime-extractor-jd57c","openshift-cluster-node-tuning-operator/tuned-pb69k","openshift-dns/dns-default-tjbxb","openshift-insights/periodic-gathering-cn27f-rccw9","openshift-kube-storage-version-migrator/migrator-66df5d496b-94228","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7","openshift-multus/multus-additional-cni-plugins-876md","kube-system/konnectivity-agent-vgcnn","openshift-image-registry/node-ca-6zmgp","openshift-multus/network-metrics-daemon-xb5xn","openshift-ovn-kubernetes/ovnkube-node-kqbrc","openshift-service-ca/service-ca-5fffc97bdf-q2bzw","kube-system/kube-apiserver-proxy-ip-10-0-135-132.ec2.internal","kube-system/global-pull-secret-syncer-lcd4g","openshift-dns/node-resolver-ndn7k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"]
Oct 30 22:06:57.091753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.091728    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:57.098735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.098711    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.100861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.100841    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.102095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102080    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44l7l\""
Oct 30 22:06:57.102329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102309    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:06:57.102329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102315    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.102329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102334    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.102521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102349    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:06:57.102521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102389    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.102521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102448    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.102792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102536    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:06:57.102792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102546    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.102792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102550    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8wxtm\""
Oct 30 22:06:57.102792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102551    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.102792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102592    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggpvh\""
Oct 30 22:06:57.102792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.102642    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:06:57.109649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.109630    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:06:57.109826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.109806    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k6z4j\""
Oct 30 22:06:57.109826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.109819    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.109826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.109831    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pwrc4\""
Oct 30 22:06:57.109975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.109808    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.109975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.109824    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:06:57.109975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.109820    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:06:57.111067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111055    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7kw8g\""
Oct 30 22:06:57.111228 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111203    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.111383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111362    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.111490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111414    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn9g8\""
Oct 30 22:06:57.111490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111442    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.111490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111443    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bf7kf\""
Oct 30 22:06:57.111490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111473    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g5ww4\""
Oct 30 22:06:57.111830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111491    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.111830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111626    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5qz2\""
Oct 30 22:06:57.111830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111819    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:06:57.111984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111968    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:06:57.112030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.111994    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:06:57.112030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112005    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:06:57.112030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112024    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:06:57.112168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112034    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.112168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112040    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.112168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112076    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:06:57.112168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112076    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:06:57.112168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112111    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xwfpf\""
Oct 30 22:06:57.112168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112155    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.112461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112199    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:06:57.112461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112254    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:06:57.112461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112261    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:06:57.112461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112361    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.112461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112389    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tjmjw\""
Oct 30 22:06:57.112461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112417    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112464    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112474    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112482    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112486    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112521    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112525    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nfwzj\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112488    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112541    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112574    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112617    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112631    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112717    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rrq2z\""
Oct 30 22:06:57.112732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112731    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:06:57.113168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112903    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z6gtt\""
Oct 30 22:06:57.113168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.112955    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.113168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.113025    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.113168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.113104    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vjfsq\""
Oct 30 22:06:57.113168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.113152    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-htsvp\""
Oct 30 22:06:57.122332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.122313    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:06:57.122402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.122322    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26rrv\""
Oct 30 22:06:57.143641 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.143608    4581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:57.158952 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.158929    4581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:06:57.163202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.163188    4581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:57.163269 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.163248    4581 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-132.ec2.internal"
Oct 30 22:06:57.198271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.198242    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-kubelet\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.198407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.198276    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-system-cni-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.198407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.198295    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.198501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.198407    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f749f836-83b3-4266-95d7-390a8c19f113-crio-socket\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.198501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.198440    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plcd4\" (UniqueName: \"kubernetes.io/projected/3c776c71-a37e-48b7-8a6b-e77542ab4bf1-kube-api-access-plcd4\") pod \"migrator-66df5d496b-94228\" (UID: \"3c776c71-a37e-48b7-8a6b-e77542ab4bf1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"
Oct 30 22:06:57.198501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.198460    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-env-overrides\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.198589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.198538    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.198975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.198948    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.199044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199014    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.199093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199056    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8c698726-813c-4953-92df-8cf3917d8bc6-signing-key\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:57.199139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199092    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-dbus\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:57.199139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199119    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-lib-modules\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.199271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199186    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-env-overrides\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.199473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199151    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-netns\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.199574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199496    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-etc-kubernetes\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.199574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199498    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8c698726-813c-4953-92df-8cf3917d8bc6-signing-key\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:57.199685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199612    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdr8r\" (UniqueName: \"kubernetes.io/projected/ec7d111b-c9ab-455f-acdd-a016d4f48531-kube-api-access-bdr8r\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.199685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199661    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df124a61-d687-4ef9-a732-2d5def6ef79c-tmp-dir\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:57.199834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199707    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7665c55-0b43-4318-9bad-342fce7a359d-agent-certs\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:57.199834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199747    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysconfig\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.199834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199783    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-systemd\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.199834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199826    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.200013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199856    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-kubelet-config\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:57.200013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199847    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.200013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199847    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df124a61-d687-4ef9-a732-2d5def6ef79c-tmp-dir\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:57.200013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199898    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-tmp\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.200013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199937    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-multus-daemon-config\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.200264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.199998    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.200264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200196    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnrc\" (UniqueName: \"kubernetes.io/projected/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kube-api-access-rqnrc\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.200344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200299    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f749f836-83b3-4266-95d7-390a8c19f113-data-volume\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.200344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200334    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.200412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200356    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7665c55-0b43-4318-9bad-342fce7a359d-agent-certs\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:57.200412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200392    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-ovn\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.200473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200433    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.200508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200487    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-cnibin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.200537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200450    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f749f836-83b3-4266-95d7-390a8c19f113-data-volume\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.200574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200514    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-multus-daemon-config\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.200607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200579    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c698726-813c-4953-92df-8cf3917d8bc6-signing-cabundle\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:57.200641 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200621    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.200673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200657    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df124a61-d687-4ef9-a732-2d5def6ef79c-hosts-file\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:57.200702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200664    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.200702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200640    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-tmp\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.200793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200712    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:57.201169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.200994    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:57.201514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201204    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-tuned\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.201514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201298    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdd6d\" (UniqueName: \"kubernetes.io/projected/543555d6-bbb3-4643-8198-9c7c397c07d9-kube-api-access-tdd6d\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:57.201514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201333    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85f74\" (UniqueName: \"kubernetes.io/projected/dece32b6-cb70-479d-9a12-bc2608b23129-kube-api-access-85f74\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.201514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201395    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2t72\" (UniqueName: \"kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:57.201514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201413    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c698726-813c-4953-92df-8cf3917d8bc6-signing-cabundle\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:57.201514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201443    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-run\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.201514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201486    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-tuned\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.201800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201543    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-kubelet\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.201800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201573    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-netns\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.201800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201596    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-bin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.201800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201631    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5wl\" (UniqueName: \"kubernetes.io/projected/8c698726-813c-4953-92df-8cf3917d8bc6-kube-api-access-gz5wl\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:57.201800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201715    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c86d61-12bf-4446-b98a-f9e571ad5ffb-host\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:57.201800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201760    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-modprobe-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.202013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201806    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgr5b\" (UniqueName: \"kubernetes.io/projected/10654728-36ab-4482-9fcb-775d88ab5aa0-kube-api-access-bgr5b\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:57.202013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201837    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-multus\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.202013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201884    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.202013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.201978    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnpjh\" (UniqueName: \"kubernetes.io/projected/df124a61-d687-4ef9-a732-2d5def6ef79c-kube-api-access-fnpjh\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:57.202013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202010    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:57.202157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202066    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16934604-e78f-4968-bdcd-3e48bd4f6882-cert\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:57.202205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202179    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-kubernetes\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.202263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202244    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-conf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.202293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202277    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-bin\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.202335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202310    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-config\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.202388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202375    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-cni-binary-copy\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.202451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202436    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-hostroot\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.202993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.202971    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-config\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.203550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.203481    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/099dbe30-a9b3-4166-b289-7fc8e25ff414-original-pull-secret\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:57.203779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.203749    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16934604-e78f-4968-bdcd-3e48bd4f6882-cert\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:57.203834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.203799    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dece32b6-cb70-479d-9a12-bc2608b23129-cni-binary-copy\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.203897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.203835    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cnibin\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.203897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.203885    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.203993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.203940    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4fx\" (UniqueName: \"kubernetes.io/projected/91c86d61-12bf-4446-b98a-f9e571ad5ffb-kube-api-access-qc4fx\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:57.204561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.204543    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7665c55-0b43-4318-9bad-342fce7a359d-konnectivity-ca\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:57.204624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.204580    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-systemd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.204624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.204599    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-script-lib\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.204624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.204618    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.204751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.204647    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:57.205033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205013    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-sys\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.205102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205080    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-var-lib-kubelet\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.205153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205131    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-slash\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205130    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:57.205273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205202    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-var-lib-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205241    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovnkube-script-lib\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205241    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7665c55-0b43-4318-9bad-342fce7a359d-konnectivity-ca\") pod \"konnectivity-agent-vgcnn\" (UID: \"c7665c55-0b43-4318-9bad-342fce7a359d\") " pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:06:57.205273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205253    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205302    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-node-log\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205328    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-system-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.205424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205364    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gbgm\" (UniqueName: \"kubernetes.io/projected/16934604-e78f-4968-bdcd-3e48bd4f6882-kube-api-access-5gbgm\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:57.205538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205427    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-host\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.205538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205501    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10654728-36ab-4482-9fcb-775d88ab5aa0-host-slash\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:57.205628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205544    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-log-socket\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205586    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-socket-dir-parent\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.205726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205633    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-multus-certs\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.205726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205659    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.205726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205683    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.205726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205712    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7d111b-c9ab-455f-acdd-a016d4f48531-tmp-dir\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.205880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205749    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91c86d61-12bf-4446-b98a-f9e571ad5ffb-serviceca\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:57.205880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205769    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-netd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205785    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-conf-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.205880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205801    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7d111b-c9ab-455f-acdd-a016d4f48531-config-volume\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.205880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205798    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7d111b-c9ab-455f-acdd-a016d4f48531-tmp-dir\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.205880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205835    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-systemd-units\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205856    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-etc-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.205880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205871    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-k8s-cni-cncf-io\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.206093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205898    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-binary-copy\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.206093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205928    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.206093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205950    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:57.206093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.205986    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:57.206093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206065    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91c86d61-12bf-4446-b98a-f9e571ad5ffb-serviceca\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:57.206093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206056    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovn-node-metrics-cert\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206087    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7d111b-c9ab-455f-acdd-a016d4f48531-config-volume\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206118    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxhf\" (UniqueName: \"kubernetes.io/projected/8b91ef29-ba61-4265-a756-ac5ad6ab604d-kube-api-access-ckxhf\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206146    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstdw\" (UniqueName: \"kubernetes.io/projected/f749f836-83b3-4266-95d7-390a8c19f113-kube-api-access-tstdw\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206171    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206192    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-ovn-node-metrics-cert\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206198    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqnf\" (UniqueName: \"kubernetes.io/projected/7f07db13-af88-4f66-a3ea-23b4314d6144-kube-api-access-mvqnf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206250    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-device-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206255    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206265    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cni-binary-copy\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206308    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10654728-36ab-4482-9fcb-775d88ab5aa0-iptables-alerter-script\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:57.206331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206329    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fbd\" (UniqueName: \"kubernetes.io/projected/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-kube-api-access-k8fbd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.206750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.206630    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10654728-36ab-4482-9fcb-775d88ab5aa0-iptables-alerter-script\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:57.219651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.219618    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plcd4\" (UniqueName: \"kubernetes.io/projected/3c776c71-a37e-48b7-8a6b-e77542ab4bf1-kube-api-access-plcd4\") pod \"migrator-66df5d496b-94228\" (UID: \"3c776c71-a37e-48b7-8a6b-e77542ab4bf1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94228"
Oct 30 22:06:57.219962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.219866    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdr8r\" (UniqueName: \"kubernetes.io/projected/ec7d111b-c9ab-455f-acdd-a016d4f48531-kube-api-access-bdr8r\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.220459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.220440    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdd6d\" (UniqueName: \"kubernetes.io/projected/543555d6-bbb3-4643-8198-9c7c397c07d9-kube-api-access-tdd6d\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:57.220705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.220681    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnpjh\" (UniqueName: \"kubernetes.io/projected/df124a61-d687-4ef9-a732-2d5def6ef79c-kube-api-access-fnpjh\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:57.220779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.220732    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85f74\" (UniqueName: \"kubernetes.io/projected/dece32b6-cb70-479d-9a12-bc2608b23129-kube-api-access-85f74\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.220881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.220864    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnrc\" (UniqueName: \"kubernetes.io/projected/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kube-api-access-rqnrc\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.221037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.221022    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5wl\" (UniqueName: \"kubernetes.io/projected/8c698726-813c-4953-92df-8cf3917d8bc6-kube-api-access-gz5wl\") pod \"service-ca-5fffc97bdf-q2bzw\" (UID: \"8c698726-813c-4953-92df-8cf3917d8bc6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-q2bzw"
Oct 30 22:06:57.223419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.223398    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gbgm\" (UniqueName: \"kubernetes.io/projected/16934604-e78f-4968-bdcd-3e48bd4f6882-kube-api-access-5gbgm\") pod \"ingress-canary-456zn\" (UID: \"16934604-e78f-4968-bdcd-3e48bd4f6882\") " pod="openshift-ingress-canary/ingress-canary-456zn"
Oct 30 22:06:57.223724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.223707    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4r4g\" (UniqueName: \"kubernetes.io/projected/650a07ef-80ee-44a8-8e96-b6bd104beb37-kube-api-access-r4r4g\") pod \"network-check-target-2k9k2\" (UID: \"650a07ef-80ee-44a8-8e96-b6bd104beb37\") " pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:06:57.223724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.223717    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4fx\" (UniqueName: \"kubernetes.io/projected/91c86d61-12bf-4446-b98a-f9e571ad5ffb-kube-api-access-qc4fx\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:57.223836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.223818    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fbd\" (UniqueName: \"kubernetes.io/projected/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-kube-api-access-k8fbd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.223977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.223960    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgr5b\" (UniqueName: \"kubernetes.io/projected/10654728-36ab-4482-9fcb-775d88ab5aa0-kube-api-access-bgr5b\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:57.224372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.224352    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxhf\" (UniqueName: \"kubernetes.io/projected/8b91ef29-ba61-4265-a756-ac5ad6ab604d-kube-api-access-ckxhf\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.224410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.224369    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2t72\" (UniqueName: \"kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72\") pod \"periodic-gathering-cn27f-rccw9\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") " pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:06:57.224484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.224465    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqnf\" (UniqueName: \"kubernetes.io/projected/7f07db13-af88-4f66-a3ea-23b4314d6144-kube-api-access-mvqnf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.225098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.225078    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstdw\" (UniqueName: \"kubernetes.io/projected/f749f836-83b3-4266-95d7-390a8c19f113-kube-api-access-tstdw\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.307672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307589    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-kubelet\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.307672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307625    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-netns\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.307672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307643    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-bin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.307672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307658    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c86d61-12bf-4446-b98a-f9e571ad5ffb-host\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:57.307672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307675    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-modprobe-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307690    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-multus\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307713    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307720    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-bin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307732    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-kubernetes\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307709    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-kubelet\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307720    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-netns\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307756    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-conf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307760    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-cni-multus\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307755    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c86d61-12bf-4446-b98a-f9e571ad5ffb-host\") pod \"node-ca-6zmgp\" (UID: \"91c86d61-12bf-4446-b98a-f9e571ad5ffb\") " pod="openshift-image-registry/node-ca-6zmgp"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307781    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-bin\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307801    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-kubernetes\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307814    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-bin\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307821    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-hostroot\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307843    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cnibin\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307865    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307873    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-conf\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307872    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-hostroot\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307885    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-modprobe-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307891    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-systemd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307919    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-cnibin\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307967    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.307981    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308012    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-systemd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308059    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-sys\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308074    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308087    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-var-lib-kubelet\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308088    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-sys\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308117    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-slash\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308142    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-var-lib-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308152    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-var-lib-kubelet\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308172    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308188    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-slash\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308196    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-node-log\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308210    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-var-lib-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308245    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-system-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308269    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-node-log\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308268    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-host\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308279    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308289    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-system-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308310    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-host\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308316    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f749f836-83b3-4266-95d7-390a8c19f113-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308312    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10654728-36ab-4482-9fcb-775d88ab5aa0-host-slash\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308333    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10654728-36ab-4482-9fcb-775d88ab5aa0-host-slash\") pod \"iptables-alerter-8kmq8\" (UID: \"10654728-36ab-4482-9fcb-775d88ab5aa0\") " pod="openshift-network-operator/iptables-alerter-8kmq8"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308339    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-log-socket\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308356    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-socket-dir-parent\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308374    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-multus-certs\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308417    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308407    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-log-socket\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308424    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-socket-dir-parent\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308420    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-multus-certs\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308435    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308470    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308475    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308475    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-netd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308508    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-conf-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308525    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-systemd-units\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308540    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-etc-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308507    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-cni-netd\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308554    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-k8s-cni-cncf-io\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308561    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-conf-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308581    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308593    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-etc-openvswitch\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308607    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-k8s-cni-cncf-io\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308615    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308569    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-systemd-units\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308625    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308680    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-device-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.308706    4581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308725    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-device-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308729    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-kubelet\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308748    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-var-lib-kubelet\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308755    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-system-cni-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308771    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308774    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b91ef29-ba61-4265-a756-ac5ad6ab604d-system-cni-dir\") pod \"multus-additional-cni-plugins-876md\" (UID: \"8b91ef29-ba61-4265-a756-ac5ad6ab604d\") " pod="openshift-multus/multus-additional-cni-plugins-876md"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.308797    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.808781003 +0000 UTC m=+1.847719908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : secret "metrics-daemon-secret" not found
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308808    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f749f836-83b3-4266-95d7-390a8c19f113-crio-socket\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.308782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308812    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99488cc5-1b6c-4b64-a38e-6403a080f2a1-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8z7\" (UID: \"99488cc5-1b6c-4b64-a38e-6403a080f2a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8z7"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308827    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308844    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308861    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-dbus\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308878    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-lib-modules\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308885    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f749f836-83b3-4266-95d7-390a8c19f113-crio-socket\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308902    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-netns\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308904    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-multus-cni-dir\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308924    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-etc-kubernetes\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308939    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-dbus\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308953    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysconfig\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308948    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-host-run-netns\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308976    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-systemd\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308982    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-etc-kubernetes\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309008    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysconfig\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309009    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309028    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-systemd\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309034    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.308983    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-lib-modules\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309036    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-kubelet-config\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309074    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309098    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-ovn\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309074    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/099dbe30-a9b3-4166-b289-7fc8e25ff414-kubelet-config\") pod \"global-pull-secret-syncer-lcd4g\" (UID: \"099dbe30-a9b3-4166-b289-7fc8e25ff414\") " pod="kube-system/global-pull-secret-syncer-lcd4g"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309122    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309143    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-cnibin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309159    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309163    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309160    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad9995b8-ba05-47ba-81c6-6f74c234f2c1-run-ovn\") pod \"ovnkube-node-kqbrc\" (UID: \"ad9995b8-ba05-47ba-81c6-6f74c234f2c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309158    4581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309200    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df124a61-d687-4ef9-a732-2d5def6ef79c-hosts-file\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309205    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dece32b6-cb70-479d-9a12-bc2608b23129-cnibin\") pod \"multus-l5shw\" (UID: \"dece32b6-cb70-479d-9a12-bc2608b23129\") " pod="openshift-multus/multus-l5shw"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309199    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-etc-sysctl-d\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.309263    4581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309274    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df124a61-d687-4ef9-a732-2d5def6ef79c-hosts-file\") pod \"node-resolver-ndn7k\" (UID: \"df124a61-d687-4ef9-a732-2d5def6ef79c\") " pod="openshift-dns/node-resolver-ndn7k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309274    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-run\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.309309    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.809290904 +0000 UTC m=+1.848229788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.310470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.309313    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f07db13-af88-4f66-a3ea-23b4314d6144-run\") pod \"tuned-pb69k\" (UID: \"7f07db13-af88-4f66-a3ea-23b4314d6144\") " pod="openshift-cluster-node-tuning-operator/tuned-pb69k"
Oct 30 22:06:57.311414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.311401    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec7d111b-c9ab-455f-acdd-a016d4f48531-metrics-tls\") pod \"dns-default-tjbxb\" (UID: \"ec7d111b-c9ab-455f-acdd-a016d4f48531\") " pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.399071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.399038    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:06:57.695269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.695238    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tjbxb"]
Oct 30 22:06:57.700936 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:06:57.700902    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7d111b_c9ab_455f_acdd_a016d4f48531.slice/crio-bcfcc24cc043fb1c5ade5e5f7db2fabb1b01b78cb924e6a4647ee14787a62e21 WatchSource:0}: Error finding container bcfcc24cc043fb1c5ade5e5f7db2fabb1b01b78cb924e6a4647ee14787a62e21: Status 404 returned error can't find the container with id bcfcc24cc043fb1c5ade5e5f7db2fabb1b01b78cb924e6a4647ee14787a62e21
Oct 30 22:06:57.703692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.703674    4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:06:57.719989 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.719968    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ndn7k_df124a61-d687-4ef9-a732-2d5def6ef79c/dns-node-resolver/0.log"
Oct 30 22:06:57.812765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.812720    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:57.812939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.812784    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:57.812939 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.812881    4581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:57.812939 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.812895    4581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.813026 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.812946    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:58.812930402 +0000 UTC m=+2.851869288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : secret "metrics-daemon-secret" not found
Oct 30 22:06:57.813026 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:57.812972    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:58.812954492 +0000 UTC m=+2.851893357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.903906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:57.903880    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:06:58.147563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:58.147527    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjbxb" event={"ID":"ec7d111b-c9ab-455f-acdd-a016d4f48531","Type":"ContainerStarted","Data":"bcfcc24cc043fb1c5ade5e5f7db2fabb1b01b78cb924e6a4647ee14787a62e21"}
Oct 30 22:06:58.320946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:58.320920    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6zmgp_91c86d61-12bf-4446-b98a-f9e571ad5ffb/node-ca/0.log"
Oct 30 22:06:58.820159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:58.820124    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:06:58.820348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:58.820176    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:06:58.820348 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:58.820325    4581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:58.820483 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:58.820326    4581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:58.820483 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:58.820458    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:00.820437527 +0000 UTC m=+4.859376414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:58.820590 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:06:58.820527    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:00.82050319 +0000 UTC m=+4.859442097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : secret "metrics-daemon-secret" not found
Oct 30 22:06:59.120955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:59.120927    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-456zn_16934604-e78f-4968-bdcd-3e48bd4f6882/serve-healthcheck-canary/0.log"
Oct 30 22:06:59.721001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:06:59.720916    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-94228_3c776c71-a37e-48b7-8a6b-e77542ab4bf1/migrator/0.log"
Oct 30 22:07:00.155261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:00.155204    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjbxb" event={"ID":"ec7d111b-c9ab-455f-acdd-a016d4f48531","Type":"ContainerStarted","Data":"b793ee42e217077d73ef58ab07b9196c92a08f933b56eb33bb151e0c5a52723b"}
Oct 30 22:07:00.155430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:00.155284    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjbxb" event={"ID":"ec7d111b-c9ab-455f-acdd-a016d4f48531","Type":"ContainerStarted","Data":"3e3ebd2f5729f07f4bc51866497423d48e829866959ef011b999e6d949bbc2c3"}
Oct 30 22:07:00.155430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:00.155370    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:07:00.835033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:00.834998    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:07:00.835440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:00.835051    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:07:00.835440 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:07:00.835145    4581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:07:00.835440 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:07:00.835149    4581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:07:00.835440 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:07:00.835209    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:04.835195599 +0000 UTC m=+8.874134476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : secret "metrics-daemon-secret" not found
Oct 30 22:07:00.835440 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:07:00.835240    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls podName:f749f836-83b3-4266-95d7-390a8c19f113 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:04.835234324 +0000 UTC m=+8.874173190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd57c" (UID: "f749f836-83b3-4266-95d7-390a8c19f113") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:07:02.936341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:02.936299    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:07:04.858819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:04.858766    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:07:04.859279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:04.858844    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:07:04.859279 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:07:04.858942    4581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:07:04.859279 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:07:04.859001    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs podName:543555d6-bbb3-4643-8198-9c7c397c07d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:12.858985363 +0000 UTC m=+16.897924242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs") pod "network-metrics-daemon-xb5xn" (UID: "543555d6-bbb3-4643-8198-9c7c397c07d9") : secret "metrics-daemon-secret" not found
Oct 30 22:07:04.861389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:04.861361    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f749f836-83b3-4266-95d7-390a8c19f113-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd57c\" (UID: \"f749f836-83b3-4266-95d7-390a8c19f113\") " pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:07:04.901936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:04.901901    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jd57c"
Oct 30 22:07:05.191728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:05.191648    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jd57c"]
Oct 30 22:07:05.197452 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:07:05.197425    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf749f836_83b3_4266_95d7_390a8c19f113.slice/crio-62dc384f82d83ce0017d9fd559fe9c0dcfbe7d9873f8de6e1a1727ff648f79ef WatchSource:0}: Error finding container 62dc384f82d83ce0017d9fd559fe9c0dcfbe7d9873f8de6e1a1727ff648f79ef: Status 404 returned error can't find the container with id 62dc384f82d83ce0017d9fd559fe9c0dcfbe7d9873f8de6e1a1727ff648f79ef
Oct 30 22:07:06.138202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:06.138125    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:07:06.138566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:06.138349    4581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:07:06.180714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:06.180670    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jd57c" event={"ID":"f749f836-83b3-4266-95d7-390a8c19f113","Type":"ContainerStarted","Data":"e211a74d4187d34068dcb3773992cabd413e292b60d677056b40a57349966bfa"}
Oct 30 22:07:06.180858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:06.180772    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jd57c" event={"ID":"f749f836-83b3-4266-95d7-390a8c19f113","Type":"ContainerStarted","Data":"d1e3f3d3172e49b0527db117b4c6a45fa7535532513f8ba7a91ac5398bcdb5e5"}
Oct 30 22:07:06.180858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:06.180794    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jd57c" event={"ID":"f749f836-83b3-4266-95d7-390a8c19f113","Type":"ContainerStarted","Data":"62dc384f82d83ce0017d9fd559fe9c0dcfbe7d9873f8de6e1a1727ff648f79ef"}
Oct 30 22:07:08.188388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:08.188347    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jd57c" event={"ID":"f749f836-83b3-4266-95d7-390a8c19f113","Type":"ContainerStarted","Data":"fc0124b6ff0768fdaae02d6fdb9ce6140841c1eb1a7206a978ed69ede83c07e8"}
Oct 30 22:07:10.035519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.035491    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6897644698-7z8kx"]
Oct 30 22:07:10.038794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.038768    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.080632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.080597    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-49l9d\""
Oct 30 22:07:10.080632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.080623    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:07:10.080855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.080689    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:07:10.080855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.080701    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:07:10.080855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.080745    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:07:10.081694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.081678    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6897644698-7z8kx"]
Oct 30 22:07:10.160984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.160957    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tjbxb"
Oct 30 22:07:10.198500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.198463    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-registry-tls\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.198718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.198522    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-trusted-ca\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.198718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.198579    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-installation-pull-secrets\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.199350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.199318    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-image-registry-private-configuration\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.199473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.199359    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-ca-trust-extracted\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.199473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.199385    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6v4\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-kube-api-access-8x6v4\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.199540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.199505    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-registry-certificates\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.199585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.199568    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-bound-sa-token\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.300941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.300849    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-bound-sa-token\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.300941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.300903    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-registry-tls\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.300941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.300932    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-trusted-ca\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.301250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.300951    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-installation-pull-secrets\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.301250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.301016    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-image-registry-private-configuration\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.301250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.301043    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-ca-trust-extracted\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.301250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.301067    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6v4\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-kube-api-access-8x6v4\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.301250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.301094    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-registry-certificates\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.301878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.301787    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-ca-trust-extracted\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.302077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.302049    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-registry-certificates\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.302205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.302186    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-trusted-ca\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.303633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.303613    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-registry-tls\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.303745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.303726    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-image-registry-private-configuration\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.303787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.303766    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-installation-pull-secrets\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.317604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.317574    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-bound-sa-token\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.317734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.317715    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6v4\" (UniqueName: \"kubernetes.io/projected/ad9e8db5-c50a-4e8f-be5e-11c41d078f13-kube-api-access-8x6v4\") pod \"image-registry-6897644698-7z8kx\" (UID: \"ad9e8db5-c50a-4e8f-be5e-11c41d078f13\") " pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.349747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.349698    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:10.667055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:10.667020    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6897644698-7z8kx"]
Oct 30 22:07:10.673110 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:07:10.673083    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9e8db5_c50a_4e8f_be5e_11c41d078f13.slice/crio-611ee7077ed2a740d128b689204f458c259f716f34a145e50826a63f6342380d WatchSource:0}: Error finding container 611ee7077ed2a740d128b689204f458c259f716f34a145e50826a63f6342380d: Status 404 returned error can't find the container with id 611ee7077ed2a740d128b689204f458c259f716f34a145e50826a63f6342380d
Oct 30 22:07:10.815653 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:07:10.815617    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:07:11.135799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:11.135777    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqbrc"
Oct 30 22:07:11.200535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:11.200498    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6897644698-7z8kx" event={"ID":"ad9e8db5-c50a-4e8f-be5e-11c41d078f13","Type":"ContainerStarted","Data":"1f57688ae2c1742820507971cdde4228ad35d635fa804433d1166962fab527cd"}
Oct 30 22:07:11.200730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:11.200580    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6897644698-7z8kx" event={"ID":"ad9e8db5-c50a-4e8f-be5e-11c41d078f13","Type":"ContainerStarted","Data":"611ee7077ed2a740d128b689204f458c259f716f34a145e50826a63f6342380d"}
Oct 30 22:07:11.200730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:11.200701    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:11.237658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:11.237607    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6897644698-7z8kx" podStartSLOduration=1.237592601 podStartE2EDuration="1.237592601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:11.236596748 +0000 UTC m=+15.275535633" watchObservedRunningTime="2026-10-30 22:07:11.237592601 +0000 UTC m=+15.276531484"
Oct 30 22:07:12.923195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:12.923151    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:07:12.925624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:12.925604    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543555d6-bbb3-4643-8198-9c7c397c07d9-metrics-certs\") pod \"network-metrics-daemon-xb5xn\" (UID: \"543555d6-bbb3-4643-8198-9c7c397c07d9\") " pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:07:12.992966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:12.992926    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xb5xn"
Oct 30 22:07:13.280633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:13.280554    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xb5xn"]
Oct 30 22:07:13.286322 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:07:13.286292    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543555d6_bbb3_4643_8198_9c7c397c07d9.slice/crio-a9f18caf9eb1751b84bfef325a7d0b9844a047035daf42a8cdf0632852e4f9c8 WatchSource:0}: Error finding container a9f18caf9eb1751b84bfef325a7d0b9844a047035daf42a8cdf0632852e4f9c8: Status 404 returned error can't find the container with id a9f18caf9eb1751b84bfef325a7d0b9844a047035daf42a8cdf0632852e4f9c8
Oct 30 22:07:14.211771 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:14.211735    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xb5xn" event={"ID":"543555d6-bbb3-4643-8198-9c7c397c07d9","Type":"ContainerStarted","Data":"f48cd4e062b0ab1df3595478ebf8ce14ff4907baad8731f8bb914b21aabd0002"}
Oct 30 22:07:14.212141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:14.211802    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xb5xn" event={"ID":"543555d6-bbb3-4643-8198-9c7c397c07d9","Type":"ContainerStarted","Data":"a9f18caf9eb1751b84bfef325a7d0b9844a047035daf42a8cdf0632852e4f9c8"}
Oct 30 22:07:15.216296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:15.216266    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xb5xn" event={"ID":"543555d6-bbb3-4643-8198-9c7c397c07d9","Type":"ContainerStarted","Data":"2fa0a6f19806f4b52d2e9ea4617100be806e336d50b0cf26f86fca2397f9f359"}
Oct 30 22:07:17.339196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.339167    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nfjcp"]
Oct 30 22:07:17.344152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.344130    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.365492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.365462    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 22:07:17.365492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.365475    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:07:17.365492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.365477    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:07:17.365704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.365504    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 22:07:17.373718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.373690    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 22:07:17.373718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.373717    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-btsx8\""
Oct 30 22:07:17.373901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.373874    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 22:07:17.453371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453335    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-root\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.453371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453376    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-sys\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.453601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453399    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-tls\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.453601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453415    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vskg\" (UniqueName: \"kubernetes.io/projected/8d14e9a0-d597-4d44-a61e-fb267d348a54-kube-api-access-2vskg\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.453601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453489    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-textfile\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.453601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453592    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-wtmp\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.453759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453739    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d14e9a0-d597-4d44-a61e-fb267d348a54-metrics-client-ca\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.453793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453781    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.453824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.453805    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.554995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.554958    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d14e9a0-d597-4d44-a61e-fb267d348a54-metrics-client-ca\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.554995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.554999    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555019    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555038    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-root\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555063    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-sys\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555081    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-tls\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555104    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vskg\" (UniqueName: \"kubernetes.io/projected/8d14e9a0-d597-4d44-a61e-fb267d348a54-kube-api-access-2vskg\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555119    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-root\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555137    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-sys\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555130    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-textfile\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555273    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-wtmp\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555420    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-wtmp\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555589    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-textfile\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555729    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d14e9a0-d597-4d44-a61e-fb267d348a54-metrics-client-ca\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.555816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.555784    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-accelerators-collector-config\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.557692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.557673    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-tls\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.557733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.557713    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d14e9a0-d597-4d44-a61e-fb267d348a54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.572114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.572081    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vskg\" (UniqueName: \"kubernetes.io/projected/8d14e9a0-d597-4d44-a61e-fb267d348a54-kube-api-access-2vskg\") pod \"node-exporter-nfjcp\" (UID: \"8d14e9a0-d597-4d44-a61e-fb267d348a54\") " pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.659538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:17.659507    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nfjcp"
Oct 30 22:07:17.680854 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:07:17.680754    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d14e9a0_d597_4d44_a61e_fb267d348a54.slice/crio-06a879a79e54da014b6365f45650714a9823c877d24fd14ffbef9a7081383340 WatchSource:0}: Error finding container 06a879a79e54da014b6365f45650714a9823c877d24fd14ffbef9a7081383340: Status 404 returned error can't find the container with id 06a879a79e54da014b6365f45650714a9823c877d24fd14ffbef9a7081383340
Oct 30 22:07:18.228676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:18.228636    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nfjcp" event={"ID":"8d14e9a0-d597-4d44-a61e-fb267d348a54","Type":"ContainerStarted","Data":"06a879a79e54da014b6365f45650714a9823c877d24fd14ffbef9a7081383340"}
Oct 30 22:07:19.233408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.233370    4581 generic.go:356] "Generic (PLEG): container finished" podID="8d14e9a0-d597-4d44-a61e-fb267d348a54" containerID="2e7665f2f0eefb3efee4c4b537ec71a35e290880b76e310380e348c11cf97dca" exitCode=0
Oct 30 22:07:19.233795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.233416    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nfjcp" event={"ID":"8d14e9a0-d597-4d44-a61e-fb267d348a54","Type":"ContainerDied","Data":"2e7665f2f0eefb3efee4c4b537ec71a35e290880b76e310380e348c11cf97dca"}
Oct 30 22:07:19.577482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.577399    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84fd7577f5-njz8x"]
Oct 30 22:07:19.580990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.580969    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.593711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.593684    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 22:07:19.593885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.593865    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 22:07:19.593968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.593959    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l52b5\""
Oct 30 22:07:19.594019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.593971    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 22:07:19.594019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.593991    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 22:07:19.594019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.594006    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 22:07:19.594127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.594010    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3d9jtg4e24al3\""
Oct 30 22:07:19.606113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.606089    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84fd7577f5-njz8x"]
Oct 30 22:07:19.673630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673590    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6551a999-524e-4039-9d02-043975af6d73-metrics-client-ca\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.673630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673635    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpmt\" (UniqueName: \"kubernetes.io/projected/6551a999-524e-4039-9d02-043975af6d73-kube-api-access-jhpmt\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.673834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673720    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.673834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673760    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.673834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673821    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-grpc-tls\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.673925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673854    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.673925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673875    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6551a999-524e-4039-9d02-043975af6d73-tmp\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.673984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673935    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.674016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.673980    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-tls\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.774983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.774951    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6551a999-524e-4039-9d02-043975af6d73-metrics-client-ca\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.774995    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpmt\" (UniqueName: \"kubernetes.io/projected/6551a999-524e-4039-9d02-043975af6d73-kube-api-access-jhpmt\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775019    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775038    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775058    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-grpc-tls\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775174    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775252    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6551a999-524e-4039-9d02-043975af6d73-tmp\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775297    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775327    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-tls\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775652    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6551a999-524e-4039-9d02-043975af6d73-tmp\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.775868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.775839    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6551a999-524e-4039-9d02-043975af6d73-metrics-client-ca\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.778103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.778081    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.778194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.778119    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.778486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.778460    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.778524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.778464    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.778553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.778538    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-thanos-querier-tls\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.778628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.778612    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6551a999-524e-4039-9d02-043975af6d73-secret-grpc-tls\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.791912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.791885    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpmt\" (UniqueName: \"kubernetes.io/projected/6551a999-524e-4039-9d02-043975af6d73-kube-api-access-jhpmt\") pod \"thanos-querier-84fd7577f5-njz8x\" (UID: \"6551a999-524e-4039-9d02-043975af6d73\") " pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:19.902860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:19.902827    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:20.209698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:20.209606    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgcnn"
Oct 30 22:07:20.226402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:20.226374    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84fd7577f5-njz8x"]
Oct 30 22:07:20.231554 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:07:20.231528    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6551a999_524e_4039_9d02_043975af6d73.slice/crio-53e30886fd2946e92ba7c41bd87cd7b806a6fd6666bfb9b4c147d6330c2a3c23 WatchSource:0}: Error finding container 53e30886fd2946e92ba7c41bd87cd7b806a6fd6666bfb9b4c147d6330c2a3c23: Status 404 returned error can't find the container with id 53e30886fd2946e92ba7c41bd87cd7b806a6fd6666bfb9b4c147d6330c2a3c23
Oct 30 22:07:20.238632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:20.238598    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nfjcp" event={"ID":"8d14e9a0-d597-4d44-a61e-fb267d348a54","Type":"ContainerStarted","Data":"a9db9cb71f49533ee7ac7f98b31adcc1a7b90a2d08603f9b90522141390103da"}
Oct 30 22:07:20.239105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:20.238654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nfjcp" event={"ID":"8d14e9a0-d597-4d44-a61e-fb267d348a54","Type":"ContainerStarted","Data":"cc9d8dc0f23e077ba8514191343e3c1559c4168798e977e9a9c616d282f712fe"}
Oct 30 22:07:20.240339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:20.240316    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x" event={"ID":"6551a999-524e-4039-9d02-043975af6d73","Type":"ContainerStarted","Data":"53e30886fd2946e92ba7c41bd87cd7b806a6fd6666bfb9b4c147d6330c2a3c23"}
Oct 30 22:07:20.275592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:20.275539    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nfjcp" podStartSLOduration=2.256914589 podStartE2EDuration="3.275525232s" totalImagesPullingTime="942.864825ms" totalInitContainerRuntime="75.745818ms" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:20.274747754 +0000 UTC m=+24.313686639" watchObservedRunningTime="2026-10-30 22:07:20.275525232 +0000 UTC m=+24.314464114"
Oct 30 22:07:22.249819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:22.249786    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x" event={"ID":"6551a999-524e-4039-9d02-043975af6d73","Type":"ContainerStarted","Data":"31be51dcb43214081ab56168be7babc18671a7cd1082d647954e09eff1480be5"}
Oct 30 22:07:23.256251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:23.256206    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x" event={"ID":"6551a999-524e-4039-9d02-043975af6d73","Type":"ContainerStarted","Data":"b9a27789be8b00d2c7b6c772f576afd5ea399d0cd666486993459586089ed334"}
Oct 30 22:07:23.256539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:23.256286    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x" event={"ID":"6551a999-524e-4039-9d02-043975af6d73","Type":"ContainerStarted","Data":"1c7ffeda1649ed3003a15c46ca07d06cd3b244dd3b4342c7a931eaa26ca87308"}
Oct 30 22:07:23.256539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:23.256314    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x" event={"ID":"6551a999-524e-4039-9d02-043975af6d73","Type":"ContainerStarted","Data":"ee551b815b42543e36b69f7387c1ab8cbe6e73d3ef31aad6f4376c011bb48120"}
Oct 30 22:07:24.262256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:24.262209    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x" event={"ID":"6551a999-524e-4039-9d02-043975af6d73","Type":"ContainerStarted","Data":"a60d0aba1e870e491f822944a2a46b0d9b53094aab6cd8f48d8e39da6d98fe3e"}
Oct 30 22:07:24.262724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:24.262287    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x" event={"ID":"6551a999-524e-4039-9d02-043975af6d73","Type":"ContainerStarted","Data":"7335358fe7ef46590592e3b74dcb3caf6d11872e07c97704175de1119c934cc2"}
Oct 30 22:07:24.262724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:24.262484    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:24.320592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:24.320540    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x" podStartSLOduration=2.5517245859999997 podStartE2EDuration="5.320525454s" totalImagesPullingTime="2.768800868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:24.310877023 +0000 UTC m=+28.349815909" watchObservedRunningTime="2026-10-30 22:07:24.320525454 +0000 UTC m=+28.359464339"
Oct 30 22:07:30.271116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.271089    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84fd7577f5-njz8x"
Oct 30 22:07:30.857463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.857428    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-plmfm"]
Oct 30 22:07:30.860069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.860049    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:30.872161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.872143    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 22:07:30.875440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.875426    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-8fl8p\""
Oct 30 22:07:30.891201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.891174    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-plmfm"]
Oct 30 22:07:30.960851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.960813    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkktr\" (UniqueName: \"kubernetes.io/projected/3b303815-9277-4c80-9f8b-14fed36152b3-kube-api-access-tkktr\") pod \"downloads-6dd789c8b7-plmfm\" (UID: \"3b303815-9277-4c80-9f8b-14fed36152b3\") " pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:30.961031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.960864    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b303815-9277-4c80-9f8b-14fed36152b3-tmp\") pod \"downloads-6dd789c8b7-plmfm\" (UID: \"3b303815-9277-4c80-9f8b-14fed36152b3\") " pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:30.972601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:30.972579    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 22:07:31.062297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:31.062266    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkktr\" (UniqueName: \"kubernetes.io/projected/3b303815-9277-4c80-9f8b-14fed36152b3-kube-api-access-tkktr\") pod \"downloads-6dd789c8b7-plmfm\" (UID: \"3b303815-9277-4c80-9f8b-14fed36152b3\") " pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:31.062451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:31.062306    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b303815-9277-4c80-9f8b-14fed36152b3-tmp\") pod \"downloads-6dd789c8b7-plmfm\" (UID: \"3b303815-9277-4c80-9f8b-14fed36152b3\") " pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:31.062630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:31.062615    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b303815-9277-4c80-9f8b-14fed36152b3-tmp\") pod \"downloads-6dd789c8b7-plmfm\" (UID: \"3b303815-9277-4c80-9f8b-14fed36152b3\") " pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:31.078946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:31.078919    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkktr\" (UniqueName: \"kubernetes.io/projected/3b303815-9277-4c80-9f8b-14fed36152b3-kube-api-access-tkktr\") pod \"downloads-6dd789c8b7-plmfm\" (UID: \"3b303815-9277-4c80-9f8b-14fed36152b3\") " pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:31.170457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:31.170387    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:31.485900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:31.485819    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-plmfm"]
Oct 30 22:07:31.491282 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:07:31.491256    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b303815_9277_4c80_9f8b_14fed36152b3.slice/crio-44ee96622c75c61cc08b7678afe8fbe8460524b93c77e5b9a35daf9bf6b03f1c WatchSource:0}: Error finding container 44ee96622c75c61cc08b7678afe8fbe8460524b93c77e5b9a35daf9bf6b03f1c: Status 404 returned error can't find the container with id 44ee96622c75c61cc08b7678afe8fbe8460524b93c77e5b9a35daf9bf6b03f1c
Oct 30 22:07:32.209149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:32.209120    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6897644698-7z8kx"
Oct 30 22:07:32.290752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:32.290717    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-plmfm" event={"ID":"3b303815-9277-4c80-9f8b-14fed36152b3","Type":"ContainerStarted","Data":"44ee96622c75c61cc08b7678afe8fbe8460524b93c77e5b9a35daf9bf6b03f1c"}
Oct 30 22:07:32.939658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:32.939629    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2k9k2"
Oct 30 22:07:40.402576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.402491    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-844db4895f-nlwmp"]
Oct 30 22:07:40.405115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.405093    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.425801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.425772    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 22:07:40.425957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.425820    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 22:07:40.425957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.425934    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 22:07:40.432883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.432858    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-serving-cert\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.433014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.432898    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-oauth-serving-cert\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.433014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.432935    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-oauth-config\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.433097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.433012    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-config\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.433097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.433082    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-tmp\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.433198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.433179    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-service-ca\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.433289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.433272    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgw4g\" (UniqueName: \"kubernetes.io/projected/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-kube-api-access-dgw4g\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.435420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.435402    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 22:07:40.435420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.435404    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4kts8\""
Oct 30 22:07:40.435551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.435467    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 22:07:40.446191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.446166    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-844db4895f-nlwmp"]
Oct 30 22:07:40.533620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.533579    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-serving-cert\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.533620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.533625    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-oauth-serving-cert\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.533867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.533658    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-oauth-config\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.533867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.533681    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-config\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.533867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.533726    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-tmp\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.533867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.533770    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-service-ca\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.533867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.533806    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgw4g\" (UniqueName: \"kubernetes.io/projected/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-kube-api-access-dgw4g\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.534514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.534449    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-tmp\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.534836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.534812    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-oauth-serving-cert\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.534932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.534812    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-config\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.534932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.534852    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-service-ca\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.536648 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.536624    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-oauth-config\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.537242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.537208    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-serving-cert\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.555396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.555367    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgw4g\" (UniqueName: \"kubernetes.io/projected/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-kube-api-access-dgw4g\") pod \"console-844db4895f-nlwmp\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") " pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.717804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:40.717720    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:40.816540 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:07:40.816507    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:07:41.015275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:41.015167    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-844db4895f-nlwmp"]
Oct 30 22:07:41.020988 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:07:41.020961    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a192c4_a5ce_4bfc_bd8c_8efbb25981d4.slice/crio-0944b0add07e3d56a9b9db99b4a28dfbe517ba06843b83dddc2feff31493062f WatchSource:0}: Error finding container 0944b0add07e3d56a9b9db99b4a28dfbe517ba06843b83dddc2feff31493062f: Status 404 returned error can't find the container with id 0944b0add07e3d56a9b9db99b4a28dfbe517ba06843b83dddc2feff31493062f
Oct 30 22:07:41.329275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:41.329177    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844db4895f-nlwmp" event={"ID":"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4","Type":"ContainerStarted","Data":"0944b0add07e3d56a9b9db99b4a28dfbe517ba06843b83dddc2feff31493062f"}
Oct 30 22:07:44.343933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:44.343845    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844db4895f-nlwmp" event={"ID":"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4","Type":"ContainerStarted","Data":"86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33"}
Oct 30 22:07:44.380474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:44.380410    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-844db4895f-nlwmp" podStartSLOduration=1.475526309 podStartE2EDuration="4.380392138s" totalImagesPullingTime="2.904865829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:44.379854785 +0000 UTC m=+48.418793683" watchObservedRunningTime="2026-10-30 22:07:44.380392138 +0000 UTC m=+48.419331028"
Oct 30 22:07:47.356596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:47.356550    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-plmfm" event={"ID":"3b303815-9277-4c80-9f8b-14fed36152b3","Type":"ContainerStarted","Data":"c588724841a3569a60e54abb93d197c1d3be8822b41dfc2a796b007bd7b7926e"}
Oct 30 22:07:47.357085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:47.356793    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:47.371674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:47.371635    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-plmfm"
Oct 30 22:07:47.412475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:47.412418    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-plmfm" podStartSLOduration=1.8419176739999998 podStartE2EDuration="17.412402642s" totalImagesPullingTime="15.570484968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:47.402667969 +0000 UTC m=+51.441606855" watchObservedRunningTime="2026-10-30 22:07:47.412402642 +0000 UTC m=+51.451341519"
Oct 30 22:07:50.184680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.184641    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db65c497c-t9zcq"]
Oct 30 22:07:50.215888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.215856    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.217202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.217174    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db65c497c-t9zcq"]
Oct 30 22:07:50.257392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.257361    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 22:07:50.321005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.320969    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-oauth-serving-cert\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.321177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.321016    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-service-ca\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.321177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.321095    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-console-config\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.321314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.321170    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4zh\" (UniqueName: \"kubernetes.io/projected/826855fb-1dfd-4482-8658-0c88ae6d2395-kube-api-access-4b4zh\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.321314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.321226    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-serving-cert\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.321408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.321315    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826855fb-1dfd-4482-8658-0c88ae6d2395-tmp\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.321408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.321358    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-trusted-ca-bundle\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.321506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.321419    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-oauth-config\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422196    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-console-config\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422280    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4zh\" (UniqueName: \"kubernetes.io/projected/826855fb-1dfd-4482-8658-0c88ae6d2395-kube-api-access-4b4zh\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422316    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-serving-cert\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422339    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826855fb-1dfd-4482-8658-0c88ae6d2395-tmp\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422361    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-trusted-ca-bundle\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422461    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-oauth-config\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422544    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-oauth-serving-cert\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422573    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-service-ca\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.422818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.422714    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826855fb-1dfd-4482-8658-0c88ae6d2395-tmp\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.423248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.423203    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-service-ca\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.423415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.423361    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-trusted-ca-bundle\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.423536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.423462    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-oauth-serving-cert\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.423916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.423897    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-console-config\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.425735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.425711    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-serving-cert\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.425829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.425716    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-oauth-config\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.439768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.439709    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4zh\" (UniqueName: \"kubernetes.io/projected/826855fb-1dfd-4482-8658-0c88ae6d2395-kube-api-access-4b4zh\") pod \"console-5db65c497c-t9zcq\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") " pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.542974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.542931    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:07:50.718888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.718797    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:50.718888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.718867    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:50.724293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.724265    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:50.860075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:50.860034    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db65c497c-t9zcq"]
Oct 30 22:07:50.867064 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:07:50.867036    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826855fb_1dfd_4482_8658_0c88ae6d2395.slice/crio-7ce73f7901f7fc85627fba0025b142822f78c19afdb55ca05dc0d2804173dfee WatchSource:0}: Error finding container 7ce73f7901f7fc85627fba0025b142822f78c19afdb55ca05dc0d2804173dfee: Status 404 returned error can't find the container with id 7ce73f7901f7fc85627fba0025b142822f78c19afdb55ca05dc0d2804173dfee
Oct 30 22:07:51.374589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:51.374548    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db65c497c-t9zcq" event={"ID":"826855fb-1dfd-4482-8658-0c88ae6d2395","Type":"ContainerStarted","Data":"76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26"}
Oct 30 22:07:51.375111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:51.374614    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db65c497c-t9zcq" event={"ID":"826855fb-1dfd-4482-8658-0c88ae6d2395","Type":"ContainerStarted","Data":"7ce73f7901f7fc85627fba0025b142822f78c19afdb55ca05dc0d2804173dfee"}
Oct 30 22:07:51.379344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:51.379315    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:07:51.410891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:07:51.410807    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5db65c497c-t9zcq" podStartSLOduration=1.410791054 podStartE2EDuration="1.410791054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:51.409598201 +0000 UTC m=+55.448537087" watchObservedRunningTime="2026-10-30 22:07:51.410791054 +0000 UTC m=+55.449729938"
Oct 30 22:08:00.543342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:00.543290    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:08:00.543827 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:00.543376    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:08:00.548072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:00.548048    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:08:01.417828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:01.417800    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:08:01.478431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:01.478405    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-844db4895f-nlwmp"]
Oct 30 22:08:10.814035 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:08:10.813993    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:08:18.468055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:18.468020    4581 generic.go:356] "Generic (PLEG): container finished" podID="7751bc9b-02bb-4854-959e-0cf6f23cb547" containerID="7a2895132c7dc1628bd4190aaf8d40bf0d1d8c740cc5801331ac72ec83da67db" exitCode=0
Oct 30 22:08:18.468055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:18.468059    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cn27f-rccw9" event={"ID":"7751bc9b-02bb-4854-959e-0cf6f23cb547","Type":"ContainerDied","Data":"7a2895132c7dc1628bd4190aaf8d40bf0d1d8c740cc5801331ac72ec83da67db"}
Oct 30 22:08:19.607743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.607717    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:08:19.759032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.758951    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path\") pod \"7751bc9b-02bb-4854-959e-0cf6f23cb547\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") "
Oct 30 22:08:19.759173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.759077    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72\" (UniqueName: \"kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72\") pod \"7751bc9b-02bb-4854-959e-0cf6f23cb547\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") "
Oct 30 22:08:19.759173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.759115    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle\") pod \"7751bc9b-02bb-4854-959e-0cf6f23cb547\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") "
Oct 30 22:08:19.759173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.759158    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle\") pod \"7751bc9b-02bb-4854-959e-0cf6f23cb547\" (UID: \"7751bc9b-02bb-4854-959e-0cf6f23cb547\") "
Oct 30 22:08:19.759361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.759339    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path" pod "7751bc9b-02bb-4854-959e-0cf6f23cb547" (UID: "7751bc9b-02bb-4854-959e-0cf6f23cb547"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:19.759524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.759497    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle" pod "7751bc9b-02bb-4854-959e-0cf6f23cb547" (UID: "7751bc9b-02bb-4854-959e-0cf6f23cb547"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:19.759639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.759554    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle" pod "7751bc9b-02bb-4854-959e-0cf6f23cb547" (UID: "7751bc9b-02bb-4854-959e-0cf6f23cb547"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:19.761487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.761467    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72" pod "7751bc9b-02bb-4854-959e-0cf6f23cb547" (UID: "7751bc9b-02bb-4854-959e-0cf6f23cb547"). InnerVolumeSpecName "kube-api-access-b2t72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:19.859664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.859624    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2t72\" (UniqueName: \"kubernetes.io/projected/7751bc9b-02bb-4854-959e-0cf6f23cb547-kube-api-access-b2t72\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:19.859664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.859655    4581 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-service-ca-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:19.859664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.859664    4581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7751bc9b-02bb-4854-959e-0cf6f23cb547-trusted-ca-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:19.859664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:19.859673    4581 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7751bc9b-02bb-4854-959e-0cf6f23cb547-archives-path\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:20.475052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:20.475013    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cn27f-rccw9" event={"ID":"7751bc9b-02bb-4854-959e-0cf6f23cb547","Type":"ContainerDied","Data":"2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473"}
Oct 30 22:08:20.475052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:20.475055    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cn27f-rccw9"
Oct 30 22:08:20.475268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:20.475088    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d64a251ffb120d35c5ae2b7289389c7d52e52e0f058279b063515be9a4a9473"
Oct 30 22:08:26.498593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.498534    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-844db4895f-nlwmp" podUID="f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" containerName="console" containerID="cri-o://86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33" gracePeriod=15
Oct 30 22:08:26.768083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.768009    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:08:26.912743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.912704    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-kube-api-access-dgw4g\" (UniqueName: \"kubernetes.io/projected/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-kube-api-access-dgw4g\") pod \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") "
Oct 30 22:08:26.912938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.912757    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-oauth-serving-cert\") pod \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") "
Oct 30 22:08:26.912938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.912780    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-oauth-config\") pod \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") "
Oct 30 22:08:26.912938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.912815    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-tmp\") pod \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") "
Oct 30 22:08:26.912938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.912846    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-config\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-config\") pod \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") "
Oct 30 22:08:26.912938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.912892    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-serving-cert\") pod \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") "
Oct 30 22:08:26.912938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.912931    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-service-ca\") pod \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\" (UID: \"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4\") "
Oct 30 22:08:26.913337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.913202    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-tmp" pod "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" (UID: "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:26.913419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.913386    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-config" pod "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" (UID: "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:26.913479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.913418    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-service-ca" pod "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" (UID: "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:26.913525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.913478    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-oauth-serving-cert" pod "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" (UID: "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:26.915415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.915391    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-serving-cert" pod "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" (UID: "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:26.915585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.915557    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-oauth-config" pod "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" (UID: "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:26.915675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:26.915588    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-kube-api-access-dgw4g" pod "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" (UID: "f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4"). InnerVolumeSpecName "kube-api-access-dgw4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:27.013986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.013946    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgw4g\" (UniqueName: \"kubernetes.io/projected/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-kube-api-access-dgw4g\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:27.013986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.013977    4581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-oauth-serving-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:27.013986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.013987    4581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-oauth-config\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:27.014204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.013998    4581 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-tmp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:27.014204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.014009    4581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-config\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:27.014204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.014018    4581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-console-serving-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:27.014204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.014025    4581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:08:27.499169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.499135    4581 generic.go:356] "Generic (PLEG): container finished" podID="f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" containerID="86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33" exitCode=0
Oct 30 22:08:27.499610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.499204    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844db4895f-nlwmp" event={"ID":"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4","Type":"ContainerDied","Data":"86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33"}
Oct 30 22:08:27.499610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.499182    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844db4895f-nlwmp"
Oct 30 22:08:27.499610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.499259    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844db4895f-nlwmp" event={"ID":"f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4","Type":"ContainerDied","Data":"0944b0add07e3d56a9b9db99b4a28dfbe517ba06843b83dddc2feff31493062f"}
Oct 30 22:08:27.499610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.499278    4581 scope.go:122] "RemoveContainer" containerID="86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33"
Oct 30 22:08:27.507445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.507429    4581 scope.go:122] "RemoveContainer" containerID="86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33"
Oct 30 22:08:27.507724 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:08:27.507704    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33\": container with ID starting with 86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33 not found: ID does not exist" containerID="86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33"
Oct 30 22:08:27.507765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.507734    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33"} err="failed to get container status \"86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33\": rpc error: code = NotFound desc = could not find container \"86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33\": container with ID starting with 86d30e606cfc5be08ef3db8c1ae75aa36ea2edd6ec56fb8f10dfa3b3f2cb6c33 not found: ID does not exist"
Oct 30 22:08:27.541483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.541461    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-844db4895f-nlwmp"]
Oct 30 22:08:27.545379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:27.545358    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-844db4895f-nlwmp"]
Oct 30 22:08:28.095991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:08:28.093865    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" path="/var/lib/kubelet/pods/f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4/volumes"
Oct 30 22:08:40.833367 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:08:40.833326    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:09:10.814453 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:09:10.814373    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:09:29.625804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.625770    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6676d75c9-mh2kk"]
Oct 30 22:09:29.626328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.626025    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" containerName="console"
Oct 30 22:09:29.626328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.626035    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" containerName="console"
Oct 30 22:09:29.626328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.626047    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7751bc9b-02bb-4854-959e-0cf6f23cb547" containerName="insights-gathering"
Oct 30 22:09:29.626328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.626053    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7751bc9b-02bb-4854-959e-0cf6f23cb547" containerName="insights-gathering"
Oct 30 22:09:29.626328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.626096    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a192c4-a5ce-4bfc-bd8c-8efbb25981d4" containerName="console"
Oct 30 22:09:29.626328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.626103    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7751bc9b-02bb-4854-959e-0cf6f23cb547" containerName="insights-gathering"
Oct 30 22:09:29.627923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.627906    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.703167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.703133    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6676d75c9-mh2kk"]
Oct 30 22:09:29.732708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.732670    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-trusted-ca-bundle\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.732708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.732711    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-config\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.732927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.732809    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7767cc7c-dedd-4ee0-b153-9076a61a563d-tmp\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.732927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.732839    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-service-ca\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.732927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.732867    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-oauth-serving-cert\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.733029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.732940    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8qs\" (UniqueName: \"kubernetes.io/projected/7767cc7c-dedd-4ee0-b153-9076a61a563d-kube-api-access-ks8qs\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.733029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.732994    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-serving-cert\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.733029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.733018    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-oauth-config\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834116    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7767cc7c-dedd-4ee0-b153-9076a61a563d-tmp\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834162    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-service-ca\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834187    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-oauth-serving-cert\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834207    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8qs\" (UniqueName: \"kubernetes.io/projected/7767cc7c-dedd-4ee0-b153-9076a61a563d-kube-api-access-ks8qs\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834267    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-serving-cert\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834297    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-oauth-config\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834338    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-trusted-ca-bundle\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834365    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-config\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.834595    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7767cc7c-dedd-4ee0-b153-9076a61a563d-tmp\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.835082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.835053    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-service-ca\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.835082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.835073    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-oauth-serving-cert\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.835299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.835073    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-config\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.835299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.835239    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7767cc7c-dedd-4ee0-b153-9076a61a563d-trusted-ca-bundle\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.837487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.837464    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-oauth-config\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.837618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.837597    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7767cc7c-dedd-4ee0-b153-9076a61a563d-console-serving-cert\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.852889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.852855    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8qs\" (UniqueName: \"kubernetes.io/projected/7767cc7c-dedd-4ee0-b153-9076a61a563d-kube-api-access-ks8qs\") pod \"console-6676d75c9-mh2kk\" (UID: \"7767cc7c-dedd-4ee0-b153-9076a61a563d\") " pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:29.938578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:29.938482    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:30.234100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:30.234016    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6676d75c9-mh2kk"]
Oct 30 22:09:30.239751 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:09:30.239726    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7767cc7c_dedd_4ee0_b153_9076a61a563d.slice/crio-3d27b510217d07668bdebbdfeed1804fecdab6a8a08408c7f7c3ebc082822243 WatchSource:0}: Error finding container 3d27b510217d07668bdebbdfeed1804fecdab6a8a08408c7f7c3ebc082822243: Status 404 returned error can't find the container with id 3d27b510217d07668bdebbdfeed1804fecdab6a8a08408c7f7c3ebc082822243
Oct 30 22:09:30.705350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:30.705309    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6676d75c9-mh2kk" event={"ID":"7767cc7c-dedd-4ee0-b153-9076a61a563d","Type":"ContainerStarted","Data":"8a6f9b026fa39b0b158d6a2413396a6b4d74abe2b9a55c46fcc5f2874b452771"}
Oct 30 22:09:30.705764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:30.705364    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6676d75c9-mh2kk" event={"ID":"7767cc7c-dedd-4ee0-b153-9076a61a563d","Type":"ContainerStarted","Data":"3d27b510217d07668bdebbdfeed1804fecdab6a8a08408c7f7c3ebc082822243"}
Oct 30 22:09:30.753878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:30.753822    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6676d75c9-mh2kk" podStartSLOduration=1.753807268 podStartE2EDuration="1.753807268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:30.743807774 +0000 UTC m=+154.782746665" watchObservedRunningTime="2026-10-30 22:09:30.753807268 +0000 UTC m=+154.792746229"
Oct 30 22:09:39.939387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:39.939323    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:39.939891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:39.939410    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:39.944173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:39.944150    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:40.742826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:40.742793    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6676d75c9-mh2kk"
Oct 30 22:09:40.810657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:09:40.810622    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db65c497c-t9zcq"]
Oct 30 22:09:40.824583 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:09:40.824542    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:10:05.832429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:05.832372    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5db65c497c-t9zcq" podUID="826855fb-1dfd-4482-8658-0c88ae6d2395" containerName="console" containerID="cri-o://76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26" gracePeriod=15
Oct 30 22:10:06.070548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.070523    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:10:06.206075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206035    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-trusted-ca-bundle\") pod \"826855fb-1dfd-4482-8658-0c88ae6d2395\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") "
Oct 30 22:10:06.206075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206078    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-oauth-config\") pod \"826855fb-1dfd-4482-8658-0c88ae6d2395\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") "
Oct 30 22:10:06.206325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206103    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826855fb-1dfd-4482-8658-0c88ae6d2395-kube-api-access-4b4zh\" (UniqueName: \"kubernetes.io/projected/826855fb-1dfd-4482-8658-0c88ae6d2395-kube-api-access-4b4zh\") pod \"826855fb-1dfd-4482-8658-0c88ae6d2395\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") "
Oct 30 22:10:06.206325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206169    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-console-config\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-console-config\") pod \"826855fb-1dfd-4482-8658-0c88ae6d2395\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") "
Oct 30 22:10:06.206325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206188    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-serving-cert\") pod \"826855fb-1dfd-4482-8658-0c88ae6d2395\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") "
Oct 30 22:10:06.206325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206209    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826855fb-1dfd-4482-8658-0c88ae6d2395-tmp\" (UniqueName: \"kubernetes.io/empty-dir/826855fb-1dfd-4482-8658-0c88ae6d2395-tmp\") pod \"826855fb-1dfd-4482-8658-0c88ae6d2395\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") "
Oct 30 22:10:06.206325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206300    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-oauth-serving-cert\") pod \"826855fb-1dfd-4482-8658-0c88ae6d2395\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") "
Oct 30 22:10:06.206611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206332    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-service-ca\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-service-ca\") pod \"826855fb-1dfd-4482-8658-0c88ae6d2395\" (UID: \"826855fb-1dfd-4482-8658-0c88ae6d2395\") "
Oct 30 22:10:06.206611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206570    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-trusted-ca-bundle" pod "826855fb-1dfd-4482-8658-0c88ae6d2395" (UID: "826855fb-1dfd-4482-8658-0c88ae6d2395"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:06.206754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206727    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826855fb-1dfd-4482-8658-0c88ae6d2395-tmp" pod "826855fb-1dfd-4482-8658-0c88ae6d2395" (UID: "826855fb-1dfd-4482-8658-0c88ae6d2395"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:06.206940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206874    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-console-config" pod "826855fb-1dfd-4482-8658-0c88ae6d2395" (UID: "826855fb-1dfd-4482-8658-0c88ae6d2395"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:06.206940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.206921    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-service-ca" pod "826855fb-1dfd-4482-8658-0c88ae6d2395" (UID: "826855fb-1dfd-4482-8658-0c88ae6d2395"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:06.207242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.207052    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-oauth-serving-cert" pod "826855fb-1dfd-4482-8658-0c88ae6d2395" (UID: "826855fb-1dfd-4482-8658-0c88ae6d2395"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:06.208474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.208445    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-oauth-config" pod "826855fb-1dfd-4482-8658-0c88ae6d2395" (UID: "826855fb-1dfd-4482-8658-0c88ae6d2395"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:06.208561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.208512    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-serving-cert" pod "826855fb-1dfd-4482-8658-0c88ae6d2395" (UID: "826855fb-1dfd-4482-8658-0c88ae6d2395"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:06.208651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.208570    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826855fb-1dfd-4482-8658-0c88ae6d2395-kube-api-access-4b4zh" pod "826855fb-1dfd-4482-8658-0c88ae6d2395" (UID: "826855fb-1dfd-4482-8658-0c88ae6d2395"). InnerVolumeSpecName "kube-api-access-4b4zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:06.307012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.306973    4581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-console-config\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.307012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.307005    4581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-serving-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.307012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.307019    4581 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826855fb-1dfd-4482-8658-0c88ae6d2395-tmp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.307290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.307032    4581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-oauth-serving-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.307290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.307043    4581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.307290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.307056    4581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826855fb-1dfd-4482-8658-0c88ae6d2395-trusted-ca-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.307290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.307067    4581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826855fb-1dfd-4482-8658-0c88ae6d2395-console-oauth-config\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.307290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.307080    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4zh\" (UniqueName: \"kubernetes.io/projected/826855fb-1dfd-4482-8658-0c88ae6d2395-kube-api-access-4b4zh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.830493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.830460    4581 generic.go:356] "Generic (PLEG): container finished" podID="826855fb-1dfd-4482-8658-0c88ae6d2395" containerID="76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26" exitCode=0
Oct 30 22:10:06.830644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.830534    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db65c497c-t9zcq"
Oct 30 22:10:06.830644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.830555    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db65c497c-t9zcq" event={"ID":"826855fb-1dfd-4482-8658-0c88ae6d2395","Type":"ContainerDied","Data":"76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26"}
Oct 30 22:10:06.830735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.830645    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db65c497c-t9zcq" event={"ID":"826855fb-1dfd-4482-8658-0c88ae6d2395","Type":"ContainerDied","Data":"7ce73f7901f7fc85627fba0025b142822f78c19afdb55ca05dc0d2804173dfee"}
Oct 30 22:10:06.830735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.830671    4581 scope.go:122] "RemoveContainer" containerID="76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26"
Oct 30 22:10:06.839594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.839450    4581 scope.go:122] "RemoveContainer" containerID="76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26"
Oct 30 22:10:06.839820 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:10:06.839744    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26\": container with ID starting with 76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26 not found: ID does not exist" containerID="76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26"
Oct 30 22:10:06.839820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.839773    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26"} err="failed to get container status \"76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26\": rpc error: code = NotFound desc = could not find container \"76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26\": container with ID starting with 76d1ad623fccfea34b5c424bd06ab348699b1e71fe444a7220de07db0b097c26 not found: ID does not exist"
Oct 30 22:10:06.874297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.874256    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db65c497c-t9zcq"]
Oct 30 22:10:06.883959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:06.883929    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5db65c497c-t9zcq"]
Oct 30 22:10:08.092605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:10:08.092568    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826855fb-1dfd-4482-8658-0c88ae6d2395" path="/var/lib/kubelet/pods/826855fb-1dfd-4482-8658-0c88ae6d2395/volumes"
Oct 30 22:10:10.815296 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:10:10.815265    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:10:40.814764 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:10:40.814675    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:11:00.586056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.586011    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"]
Oct 30 22:11:00.586642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.586439    4581 cpu_manager.go:414] "containerMap: removing container" podUID="826855fb-1dfd-4482-8658-0c88ae6d2395" containerName="console"
Oct 30 22:11:00.586642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.586457    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826855fb-1dfd-4482-8658-0c88ae6d2395" containerName="console"
Oct 30 22:11:00.586642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.586543    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="826855fb-1dfd-4482-8658-0c88ae6d2395" containerName="console"
Oct 30 22:11:00.589601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.589577    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.612998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.612965    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 22:11:00.612998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.612972    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-75jx4\""
Oct 30 22:11:00.612998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.612994    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 22:11:00.614061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.614043    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"]
Oct 30 22:11:00.696831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.696784    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.696831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.696836    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6b7b\" (UniqueName: \"kubernetes.io/projected/8a545860-1357-4871-b8d6-4a56f3c16183-kube-api-access-j6b7b\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.697039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.696923    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.798128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.798095    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.798337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.798157    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.798337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.798192    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6b7b\" (UniqueName: \"kubernetes.io/projected/8a545860-1357-4871-b8d6-4a56f3c16183-kube-api-access-j6b7b\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.798527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.798506    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.798583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.798560    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.816683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.816653    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6b7b\" (UniqueName: \"kubernetes.io/projected/8a545860-1357-4871-b8d6-4a56f3c16183-kube-api-access-j6b7b\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:00.902315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:00.902277    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:01.194958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.194875    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"]
Oct 30 22:11:01.200669 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:01.200639    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a545860_1357_4871_b8d6_4a56f3c16183.slice/crio-19b9c8c50789c851796f67a888f2c92fabc90115b2c58785c4896cedadb898d1 WatchSource:0}: Error finding container 19b9c8c50789c851796f67a888f2c92fabc90115b2c58785c4896cedadb898d1: Status 404 returned error can't find the container with id 19b9c8c50789c851796f67a888f2c92fabc90115b2c58785c4896cedadb898d1
Oct 30 22:11:01.349447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.349413    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"]
Oct 30 22:11:01.352355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.352331    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.365338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.365316    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-clpxg\""
Oct 30 22:11:01.379098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.379073    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"]
Oct 30 22:11:01.402336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.402303    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2sz\" (UniqueName: \"kubernetes.io/projected/14ef3be9-6595-4521-8aad-a8315a6add28-kube-api-access-ck2sz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.402494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.402344    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.402494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.402362    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.478837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.478757    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"]
Oct 30 22:11:01.481718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.481693    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.494362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.494334    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-d84kl\""
Oct 30 22:11:01.503819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.503787    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pfq\" (UniqueName: \"kubernetes.io/projected/8248e1e6-8e66-4f08-92e4-118f49638f02-kube-api-access-x4pfq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.503991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.503859    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.503991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.503919    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.503991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.503948    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2sz\" (UniqueName: \"kubernetes.io/projected/14ef3be9-6595-4521-8aad-a8315a6add28-kube-api-access-ck2sz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.504149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.504007    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.504149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.504033    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.504412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.504395    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.504475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.504424    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.504917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.504900    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"]
Oct 30 22:11:01.524125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.524099    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2sz\" (UniqueName: \"kubernetes.io/projected/14ef3be9-6595-4521-8aad-a8315a6add28-kube-api-access-ck2sz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.545335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.545287    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"]
Oct 30 22:11:01.548143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.548119    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.560273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.560250    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-b64xl\""
Oct 30 22:11:01.573833 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.573793    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"]
Oct 30 22:11:01.604993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.604952    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.604993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.604999    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.605544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.605027    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.605544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.605054    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.605544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.605134    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9sl\" (UniqueName: \"kubernetes.io/projected/a0e71a0d-794b-4960-b837-9f3331977139-kube-api-access-kq9sl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.605544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.605278    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pfq\" (UniqueName: \"kubernetes.io/projected/8248e1e6-8e66-4f08-92e4-118f49638f02-kube-api-access-x4pfq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.605544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.605479    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.605544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.605493    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.624882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.624845    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pfq\" (UniqueName: \"kubernetes.io/projected/8248e1e6-8e66-4f08-92e4-118f49638f02-kube-api-access-x4pfq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.648031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.647993    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"]
Oct 30 22:11:01.651060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.651034    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.663588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.663563    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-t56zv\""
Oct 30 22:11:01.664633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.664614    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:01.673821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.673655    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"]
Oct 30 22:11:01.705996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.705951    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.706184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.706056    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.706184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.706086    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s52xv\" (UniqueName: \"kubernetes.io/projected/88ca6131-c91a-4488-923a-344f0c0d3338-kube-api-access-s52xv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.706184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.706118    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.706184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.706159    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9sl\" (UniqueName: \"kubernetes.io/projected/a0e71a0d-794b-4960-b837-9f3331977139-kube-api-access-kq9sl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.706440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.706208    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.706742 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.706719    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.707017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.706996    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.725163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.724955    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9sl\" (UniqueName: \"kubernetes.io/projected/a0e71a0d-794b-4960-b837-9f3331977139-kube-api-access-kq9sl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.741817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.741743    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"]
Oct 30 22:11:01.744864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.744840    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.763808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.763777    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-ll4xv\""
Oct 30 22:11:01.768773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.768734    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"]
Oct 30 22:11:01.793328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.793299    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:01.807588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.807554    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.807698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.807610    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s52xv\" (UniqueName: \"kubernetes.io/projected/88ca6131-c91a-4488-923a-344f0c0d3338-kube-api-access-s52xv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.807771 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.807716    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.807884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.807862    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88ch\" (UniqueName: \"kubernetes.io/projected/f270a9c1-75f1-4d14-9e3b-058e0e019d41-kube-api-access-n88ch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.807981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.807955    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.808105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.807994    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.808335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.808317    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.808418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.808353    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.825297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.825269    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s52xv\" (UniqueName: \"kubernetes.io/projected/88ca6131-c91a-4488-923a-344f0c0d3338-kube-api-access-s52xv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.862548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.862516    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:01.908649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.908598    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.908838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.908659    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.908838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.908694    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88ch\" (UniqueName: \"kubernetes.io/projected/f270a9c1-75f1-4d14-9e3b-058e0e019d41-kube-api-access-n88ch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.909495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.909468    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.909748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.909670    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.928059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.927683    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88ch\" (UniqueName: \"kubernetes.io/projected/f270a9c1-75f1-4d14-9e3b-058e0e019d41-kube-api-access-n88ch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:01.965631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.965596    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:01.970277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:01.970248    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"]
Oct 30 22:11:01.976144 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:01.976114    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ef3be9_6595_4521_8aad_a8315a6add28.slice/crio-3e468ba65ac467b9ec62ff7fb3d558b5debf1754892f680774439e0eab26905f WatchSource:0}: Error finding container 3e468ba65ac467b9ec62ff7fb3d558b5debf1754892f680774439e0eab26905f: Status 404 returned error can't find the container with id 3e468ba65ac467b9ec62ff7fb3d558b5debf1754892f680774439e0eab26905f
Oct 30 22:11:02.016581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:02.016455    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq" event={"ID":"8a545860-1357-4871-b8d6-4a56f3c16183","Type":"ContainerStarted","Data":"19b9c8c50789c851796f67a888f2c92fabc90115b2c58785c4896cedadb898d1"}
Oct 30 22:11:02.018398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:02.018362    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v" event={"ID":"14ef3be9-6595-4521-8aad-a8315a6add28","Type":"ContainerStarted","Data":"3e468ba65ac467b9ec62ff7fb3d558b5debf1754892f680774439e0eab26905f"}
Oct 30 22:11:02.060811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:02.060778    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:02.095860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:02.095800    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"]
Oct 30 22:11:02.095860 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:02.095799    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8248e1e6_8e66_4f08_92e4_118f49638f02.slice/crio-11051e4067e2fb7fbdc26f71657076f0324bfaa30793e46fa5651e012d70c726 WatchSource:0}: Error finding container 11051e4067e2fb7fbdc26f71657076f0324bfaa30793e46fa5651e012d70c726: Status 404 returned error can't find the container with id 11051e4067e2fb7fbdc26f71657076f0324bfaa30793e46fa5651e012d70c726
Oct 30 22:11:02.170156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:02.170121    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"]
Oct 30 22:11:02.175504 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:02.175471    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e71a0d_794b_4960_b837_9f3331977139.slice/crio-2a4f1ec8ea798938a1ebc2398cd96f79174564e882aea55c7e4e14bef9117494 WatchSource:0}: Error finding container 2a4f1ec8ea798938a1ebc2398cd96f79174564e882aea55c7e4e14bef9117494: Status 404 returned error can't find the container with id 2a4f1ec8ea798938a1ebc2398cd96f79174564e882aea55c7e4e14bef9117494
Oct 30 22:11:02.273641 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:02.273563    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"]
Oct 30 22:11:02.279056 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:02.279021    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ca6131_c91a_4488_923a_344f0c0d3338.slice/crio-4e2b90e3ccf4c53812b04889319070443951f029f332426cd12187641b6ebdc4 WatchSource:0}: Error finding container 4e2b90e3ccf4c53812b04889319070443951f029f332426cd12187641b6ebdc4: Status 404 returned error can't find the container with id 4e2b90e3ccf4c53812b04889319070443951f029f332426cd12187641b6ebdc4
Oct 30 22:11:02.363445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:02.363413    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"]
Oct 30 22:11:02.369880 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:02.369842    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf270a9c1_75f1_4d14_9e3b_058e0e019d41.slice/crio-d6cea8f86cde4ed3dfe31a1a257b926490da6156e351e76c214feeb3d410be16 WatchSource:0}: Error finding container d6cea8f86cde4ed3dfe31a1a257b926490da6156e351e76c214feeb3d410be16: Status 404 returned error can't find the container with id d6cea8f86cde4ed3dfe31a1a257b926490da6156e351e76c214feeb3d410be16
Oct 30 22:11:03.025419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:03.025346    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp" event={"ID":"88ca6131-c91a-4488-923a-344f0c0d3338","Type":"ContainerStarted","Data":"4e2b90e3ccf4c53812b04889319070443951f029f332426cd12187641b6ebdc4"}
Oct 30 22:11:03.027169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:03.027141    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts" event={"ID":"f270a9c1-75f1-4d14-9e3b-058e0e019d41","Type":"ContainerStarted","Data":"d6cea8f86cde4ed3dfe31a1a257b926490da6156e351e76c214feeb3d410be16"}
Oct 30 22:11:03.029039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:03.029014    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks" event={"ID":"8248e1e6-8e66-4f08-92e4-118f49638f02","Type":"ContainerStarted","Data":"11051e4067e2fb7fbdc26f71657076f0324bfaa30793e46fa5651e012d70c726"}
Oct 30 22:11:03.030921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:03.030897    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms" event={"ID":"a0e71a0d-794b-4960-b837-9f3331977139","Type":"ContainerStarted","Data":"2a4f1ec8ea798938a1ebc2398cd96f79174564e882aea55c7e4e14bef9117494"}
Oct 30 22:11:05.051378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.051338    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"]
Oct 30 22:11:05.056783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.056753    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.069880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.069854    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:11:05.070029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.069992    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:11:05.079289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.079257    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j98cq\""
Oct 30 22:11:05.082547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.082429    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"]
Oct 30 22:11:05.137600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137560    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.137777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137640    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.137777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137673    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.137777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137709    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccfd601-6d98-48b5-85e3-101586b02704-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.137777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137743    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5n8\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-kube-api-access-vd5n8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.138001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137786    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.138001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137864    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.138001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137894    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.138001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.137927    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccfd601-6d98-48b5-85e3-101586b02704-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.164655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.164585    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"]
Oct 30 22:11:05.167424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.167401    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.169908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.169889    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:11:05.187128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.187083    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qjg8k\""
Oct 30 22:11:05.196012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.195938    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"]
Oct 30 22:11:05.218511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.218474    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"]
Oct 30 22:11:05.221788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.221330    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238618    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238668    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238698    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238725    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238760    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccfd601-6d98-48b5-85e3-101586b02704-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238796    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5n8\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-kube-api-access-vd5n8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238820    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238848    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238876    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238907    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.238977    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239004    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239027    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239049    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239078    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239110    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccfd601-6d98-48b5-85e3-101586b02704-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239140    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239181    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmb9w\" (UniqueName: \"kubernetes.io/projected/4718e016-9e2a-484a-ac25-b27ddd2b1f62-kube-api-access-gmb9w\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239235    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scndt\" (UniqueName: \"kubernetes.io/projected/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-kube-api-access-scndt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239281    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239336    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239543    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.239597    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.240050    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.240520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.240302    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccfd601-6d98-48b5-85e3-101586b02704-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.241881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.240541    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.249321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.249265    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"]
Oct 30 22:11:05.266447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.266090    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.266611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.266505    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccfd601-6d98-48b5-85e3-101586b02704-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.267175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.267152    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5n8\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-kube-api-access-vd5n8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.267625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.267605    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lnh72\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.287601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.287531    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7pbwq\""
Oct 30 22:11:05.339947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.339845    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.339947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.339921    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.339947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.339948    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.339978    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340009    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340045    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmb9w\" (UniqueName: \"kubernetes.io/projected/4718e016-9e2a-484a-ac25-b27ddd2b1f62-kube-api-access-gmb9w\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340074    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scndt\" (UniqueName: \"kubernetes.io/projected/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-kube-api-access-scndt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340116    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340158    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340194    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340271    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340300    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340682    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340732    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.340925    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.341467    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.341724    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.341842    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.341924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.341851    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.346329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.346283    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.356727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.356691    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.366342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.366191    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4718e016-9e2a-484a-ac25-b27ddd2b1f62-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.367151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.367045    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmb9w\" (UniqueName: \"kubernetes.io/projected/4718e016-9e2a-484a-ac25-b27ddd2b1f62-kube-api-access-gmb9w\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q\" (UID: \"4718e016-9e2a-484a-ac25-b27ddd2b1f62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.369816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.369719    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scndt\" (UniqueName: \"kubernetes.io/projected/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-kube-api-access-scndt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.376428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.376399    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:05.486741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.486583    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:05.551144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.550780    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:05.748150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.748115    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"]
Oct 30 22:11:05.753934 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:05.753895    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccfd601_6d98_48b5_85e3_101586b02704.slice/crio-327d5690f47ca8c143ac98b7c23d0af8ebded3164e2ea9d7703c2020743a659a WatchSource:0}: Error finding container 327d5690f47ca8c143ac98b7c23d0af8ebded3164e2ea9d7703c2020743a659a: Status 404 returned error can't find the container with id 327d5690f47ca8c143ac98b7c23d0af8ebded3164e2ea9d7703c2020743a659a
Oct 30 22:11:05.812383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.812283    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"]
Oct 30 22:11:05.818649 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:05.818613    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f297c2b_3ccd_4f6f_8da8_03f485b4eec4.slice/crio-71bc3040663b14945e76452d3d0508600b0e0676f5e049cdab014dd112b9a772 WatchSource:0}: Error finding container 71bc3040663b14945e76452d3d0508600b0e0676f5e049cdab014dd112b9a772: Status 404 returned error can't find the container with id 71bc3040663b14945e76452d3d0508600b0e0676f5e049cdab014dd112b9a772
Oct 30 22:11:05.906083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:05.905999    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"]
Oct 30 22:11:05.917412 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:05.917376    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4718e016_9e2a_484a_ac25_b27ddd2b1f62.slice/crio-303396519d7e0ec3f8ba47d9bad7b5ebbc2978ac04d35695fce027cdd11b45d9 WatchSource:0}: Error finding container 303396519d7e0ec3f8ba47d9bad7b5ebbc2978ac04d35695fce027cdd11b45d9: Status 404 returned error can't find the container with id 303396519d7e0ec3f8ba47d9bad7b5ebbc2978ac04d35695fce027cdd11b45d9
Oct 30 22:11:06.057885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:06.057844    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28" event={"ID":"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4","Type":"ContainerStarted","Data":"71bc3040663b14945e76452d3d0508600b0e0676f5e049cdab014dd112b9a772"}
Oct 30 22:11:06.062644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:06.062617    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q" event={"ID":"4718e016-9e2a-484a-ac25-b27ddd2b1f62","Type":"ContainerStarted","Data":"303396519d7e0ec3f8ba47d9bad7b5ebbc2978ac04d35695fce027cdd11b45d9"}
Oct 30 22:11:06.064828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:06.064798    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" event={"ID":"1ccfd601-6d98-48b5-85e3-101586b02704","Type":"ContainerStarted","Data":"327d5690f47ca8c143ac98b7c23d0af8ebded3164e2ea9d7703c2020743a659a"}
Oct 30 22:11:09.033296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.031826    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"]
Oct 30 22:11:09.037064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.036421    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.059901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.059870    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"]
Oct 30 22:11:09.070466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.070283    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"]
Oct 30 22:11:09.076119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.075775    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.076119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.075819    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.076119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.075878    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/1bd86ea7-1705-4692-af20-2da63ff666a9-kube-api-access-r8tsm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.077754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.077704    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.104844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.104813    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"]
Oct 30 22:11:09.138179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.137269    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"]
Oct 30 22:11:09.142630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.141881    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.171618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.171590    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"]
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.176895    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.176953    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5vs\" (UniqueName: \"kubernetes.io/projected/2e67d50a-4bf6-4045-8adf-50c1886284da-kube-api-access-gs5vs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.176987    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.177044    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.177067    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.177097    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.177128    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.177164    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rf4p\" (UniqueName: \"kubernetes.io/projected/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-kube-api-access-4rf4p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.177237    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/1bd86ea7-1705-4692-af20-2da63ff666a9-kube-api-access-r8tsm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.177798    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.178376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.178027    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.220161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.220122    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"]
Oct 30 22:11:09.225596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.225570    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.235559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.235481    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/1bd86ea7-1705-4692-af20-2da63ff666a9-kube-api-access-r8tsm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.248164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.248046    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"]
Oct 30 22:11:09.278399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278358    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.278592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278413    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.278592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278442    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rf4p\" (UniqueName: \"kubernetes.io/projected/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-kube-api-access-4rf4p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.278592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278474    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpsvs\" (UniqueName: \"kubernetes.io/projected/9288ccb2-6e3c-4bfc-823d-a772e11cad90-kube-api-access-kpsvs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.278592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278523    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.278592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278573    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.278592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278616    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.278944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278652    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5vs\" (UniqueName: \"kubernetes.io/projected/2e67d50a-4bf6-4045-8adf-50c1886284da-kube-api-access-gs5vs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.278944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278685    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.278944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278855    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.278944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.278870    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.279143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.279038    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.279143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.279127    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.301368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.301250    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5vs\" (UniqueName: \"kubernetes.io/projected/2e67d50a-4bf6-4045-8adf-50c1886284da-kube-api-access-gs5vs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.304406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.304293    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rf4p\" (UniqueName: \"kubernetes.io/projected/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-kube-api-access-4rf4p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.359998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.359679    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:09.379860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.379790    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.379860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.379858    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.380048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.379966    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpsvs\" (UniqueName: \"kubernetes.io/projected/9288ccb2-6e3c-4bfc-823d-a772e11cad90-kube-api-access-kpsvs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.381142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.380735    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.381142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.381023    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.402294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.402262    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:09.402465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.402336    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpsvs\" (UniqueName: \"kubernetes.io/projected/9288ccb2-6e3c-4bfc-823d-a772e11cad90-kube-api-access-kpsvs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.467724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.467283    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:09.564817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.564379    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:09.711187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.711154    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"]
Oct 30 22:11:09.762025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.761989    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"]
Oct 30 22:11:09.852360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.852266    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"]
Oct 30 22:11:09.864257 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:09.864204    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e67d50a_4bf6_4045_8adf_50c1886284da.slice/crio-45fe20b60cab11a213db165a346cb549bced5b8d70b86095e79bb7ff70eb9823 WatchSource:0}: Error finding container 45fe20b60cab11a213db165a346cb549bced5b8d70b86095e79bb7ff70eb9823: Status 404 returned error can't find the container with id 45fe20b60cab11a213db165a346cb549bced5b8d70b86095e79bb7ff70eb9823
Oct 30 22:11:09.917059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:09.916936    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"]
Oct 30 22:11:09.932442 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:09.932358    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9288ccb2_6e3c_4bfc_823d_a772e11cad90.slice/crio-a5c61186e235b38baaf0bc2b4cd4d222cc279a748e9f63c5dec5aaa2eb4cb165 WatchSource:0}: Error finding container a5c61186e235b38baaf0bc2b4cd4d222cc279a748e9f63c5dec5aaa2eb4cb165: Status 404 returned error can't find the container with id a5c61186e235b38baaf0bc2b4cd4d222cc279a748e9f63c5dec5aaa2eb4cb165
Oct 30 22:11:10.136814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:10.136774    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj" event={"ID":"1bd86ea7-1705-4692-af20-2da63ff666a9","Type":"ContainerStarted","Data":"810d07395508fa3e7310aeb87c3bf9fadd13adc6bc6d391ac222a35415b2f739"}
Oct 30 22:11:10.145781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:10.145747    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g" event={"ID":"9288ccb2-6e3c-4bfc-823d-a772e11cad90","Type":"ContainerStarted","Data":"a5c61186e235b38baaf0bc2b4cd4d222cc279a748e9f63c5dec5aaa2eb4cb165"}
Oct 30 22:11:10.164782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:10.164741    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz" event={"ID":"c1e6549a-b2b6-4156-b92d-64c5d8081ca5","Type":"ContainerStarted","Data":"ed2a4c970fd05fbcdcaaeb40fd4c009061a1a3e0ddaa240364f948df382c423e"}
Oct 30 22:11:10.167108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:10.166950    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk" event={"ID":"2e67d50a-4bf6-4045-8adf-50c1886284da","Type":"ContainerStarted","Data":"45fe20b60cab11a213db165a346cb549bced5b8d70b86095e79bb7ff70eb9823"}
Oct 30 22:11:10.945509 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:11:10.945414    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:11:16.027654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:15.990907    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:16.027654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:15.990998    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:16.027654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:15.991044    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:16.651102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:16.650302    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:16.651102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:16.650400    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:16.651102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:16.650452    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:11:17.249647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:17.249603    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q" event={"ID":"4718e016-9e2a-484a-ac25-b27ddd2b1f62","Type":"ContainerStarted","Data":"a46fe5c140473eeaa528694a19dead3201781d091f2efd9249863748c88c3636"}
Oct 30 22:11:17.256757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:17.256091    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" event={"ID":"1ccfd601-6d98-48b5-85e3-101586b02704","Type":"ContainerStarted","Data":"62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4"}
Oct 30 22:11:17.312629 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:17.311156    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q" podStartSLOduration=1.588134904 podStartE2EDuration="12.311136613s" totalImagesPullingTime="10.723001709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:17.301081715 +0000 UTC m=+261.340020602" watchObservedRunningTime="2026-10-30 22:11:17.311136613 +0000 UTC m=+261.350075497"
Oct 30 22:11:17.342384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:17.342331    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podStartSLOduration=2.109729401 podStartE2EDuration="12.342316282s" totalImagesPullingTime="10.232586881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:17.341444698 +0000 UTC m=+261.380383596" watchObservedRunningTime="2026-10-30 22:11:17.342316282 +0000 UTC m=+261.381255164"
Oct 30 22:11:17.376595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:17.376559    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:17.384187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:17.384157    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:17.552128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:17.552040    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:17.558573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:17.558542    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:18.263626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:18.263594    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:18.264710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:18.263651    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:18.264710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:18.264678    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q"
Oct 30 22:11:18.265335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:18.265288    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:11:21.306473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:21.306424    4581 generic.go:356] "Generic (PLEG): container finished" podID="8a545860-1357-4871-b8d6-4a56f3c16183" containerID="50f43b437bbef9fe6a79ffb899091c2ce5d271cffe9a315ebf24ed5d9b3f1928" exitCode=0
Oct 30 22:11:21.306940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:21.306558    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq" event={"ID":"8a545860-1357-4871-b8d6-4a56f3c16183","Type":"ContainerDied","Data":"50f43b437bbef9fe6a79ffb899091c2ce5d271cffe9a315ebf24ed5d9b3f1928"}
Oct 30 22:11:23.318586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.318548    4581 generic.go:356] "Generic (PLEG): container finished" podID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerID="d9e4d29482b810f177a09fbed11bc752826920b69766d74427a5654dc48d36db" exitCode=0
Oct 30 22:11:23.319046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.318815    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts" event={"ID":"f270a9c1-75f1-4d14-9e3b-058e0e019d41","Type":"ContainerDied","Data":"d9e4d29482b810f177a09fbed11bc752826920b69766d74427a5654dc48d36db"}
Oct 30 22:11:23.320853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.320776    4581 generic.go:356] "Generic (PLEG): container finished" podID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerID="64518dd26d184ef0e661e6aa1deb932f7d3c9e24beae8ebc1b88410c895aaff0" exitCode=0
Oct 30 22:11:23.320925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.320852    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj" event={"ID":"1bd86ea7-1705-4692-af20-2da63ff666a9","Type":"ContainerDied","Data":"64518dd26d184ef0e661e6aa1deb932f7d3c9e24beae8ebc1b88410c895aaff0"}
Oct 30 22:11:23.322966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.322940    4581 generic.go:356] "Generic (PLEG): container finished" podID="14ef3be9-6595-4521-8aad-a8315a6add28" containerID="527e18e6fb670cae2fd697c22528e5506cda3f2cf3d47173fb179ecfad3c217c" exitCode=0
Oct 30 22:11:23.323085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.323064    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v" event={"ID":"14ef3be9-6595-4521-8aad-a8315a6add28","Type":"ContainerDied","Data":"527e18e6fb670cae2fd697c22528e5506cda3f2cf3d47173fb179ecfad3c217c"}
Oct 30 22:11:23.327571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.327543    4581 generic.go:356] "Generic (PLEG): container finished" podID="a0e71a0d-794b-4960-b837-9f3331977139" containerID="8c7cc4f0187f0f74021a817946ff0de62a482d7c702e3d4c6d2167bd332a0f3b" exitCode=0
Oct 30 22:11:23.327687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.327666    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms" event={"ID":"a0e71a0d-794b-4960-b837-9f3331977139","Type":"ContainerDied","Data":"8c7cc4f0187f0f74021a817946ff0de62a482d7c702e3d4c6d2167bd332a0f3b"}
Oct 30 22:11:23.330368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.330343    4581 generic.go:356] "Generic (PLEG): container finished" podID="88ca6131-c91a-4488-923a-344f0c0d3338" containerID="0fd7b86346d59fd7493703527f2aea733711392aea4ef23272f923506652780d" exitCode=0
Oct 30 22:11:23.330472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:23.330379    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp" event={"ID":"88ca6131-c91a-4488-923a-344f0c0d3338","Type":"ContainerDied","Data":"0fd7b86346d59fd7493703527f2aea733711392aea4ef23272f923506652780d"}
Oct 30 22:11:24.339285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.338332    4581 generic.go:356] "Generic (PLEG): container finished" podID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerID="dcab3bd54dfb60f1571bf7115d48c820dc4788a8bbc8509ce627eb623808bbc5" exitCode=0
Oct 30 22:11:24.339285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.338420    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks" event={"ID":"8248e1e6-8e66-4f08-92e4-118f49638f02","Type":"ContainerDied","Data":"dcab3bd54dfb60f1571bf7115d48c820dc4788a8bbc8509ce627eb623808bbc5"}
Oct 30 22:11:24.342630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.342601    4581 generic.go:356] "Generic (PLEG): container finished" podID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerID="540008c6c688be1ab4164835d1888be68429ddfc50b74b2c74aead64cb87149a" exitCode=0
Oct 30 22:11:24.342743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.342693    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz" event={"ID":"c1e6549a-b2b6-4156-b92d-64c5d8081ca5","Type":"ContainerDied","Data":"540008c6c688be1ab4164835d1888be68429ddfc50b74b2c74aead64cb87149a"}
Oct 30 22:11:24.347242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.347196    4581 generic.go:356] "Generic (PLEG): container finished" podID="a0e71a0d-794b-4960-b837-9f3331977139" containerID="f745e254fb62bc908149fb6811986e3563c8568fab5dbff3d80821fd71e1a4ea" exitCode=0
Oct 30 22:11:24.347364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.347294    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms" event={"ID":"a0e71a0d-794b-4960-b837-9f3331977139","Type":"ContainerDied","Data":"f745e254fb62bc908149fb6811986e3563c8568fab5dbff3d80821fd71e1a4ea"}
Oct 30 22:11:24.349805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.349778    4581 generic.go:356] "Generic (PLEG): container finished" podID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerID="899ababb077941dc50dfcc7ae1e76153882cd58349be09fb08a5ac8cb790e963" exitCode=0
Oct 30 22:11:24.349899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.349884    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk" event={"ID":"2e67d50a-4bf6-4045-8adf-50c1886284da","Type":"ContainerDied","Data":"899ababb077941dc50dfcc7ae1e76153882cd58349be09fb08a5ac8cb790e963"}
Oct 30 22:11:24.359261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.359233    4581 generic.go:356] "Generic (PLEG): container finished" podID="88ca6131-c91a-4488-923a-344f0c0d3338" containerID="48ec29ecc2a62f01802553bea3ee467ef70144e63dab69abdf6233f38df7daf7" exitCode=0
Oct 30 22:11:24.359368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.359348    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp" event={"ID":"88ca6131-c91a-4488-923a-344f0c0d3338","Type":"ContainerDied","Data":"48ec29ecc2a62f01802553bea3ee467ef70144e63dab69abdf6233f38df7daf7"}
Oct 30 22:11:24.366185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.366151    4581 generic.go:356] "Generic (PLEG): container finished" podID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerID="fac3e9354786fbbfcc53c31335f8cf747cde9aa31815cc6b5a1e9b64c50dd5d3" exitCode=0
Oct 30 22:11:24.366315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.366252    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj" event={"ID":"1bd86ea7-1705-4692-af20-2da63ff666a9","Type":"ContainerDied","Data":"fac3e9354786fbbfcc53c31335f8cf747cde9aa31815cc6b5a1e9b64c50dd5d3"}
Oct 30 22:11:24.370773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.370655    4581 generic.go:356] "Generic (PLEG): container finished" podID="14ef3be9-6595-4521-8aad-a8315a6add28" containerID="4b4ef97183cbc8951c858117c4e9cc31b64cc80799e958aa4f91f2ab9f585c8a" exitCode=0
Oct 30 22:11:24.370773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.370760    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v" event={"ID":"14ef3be9-6595-4521-8aad-a8315a6add28","Type":"ContainerDied","Data":"4b4ef97183cbc8951c858117c4e9cc31b64cc80799e958aa4f91f2ab9f585c8a"}
Oct 30 22:11:24.373434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.373405    4581 generic.go:356] "Generic (PLEG): container finished" podID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerID="bfc6bab46604a5c8ac8937cda3491187e4e1a542317c96e62d7516dd77b4d7bf" exitCode=0
Oct 30 22:11:24.373543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:24.373444    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g" event={"ID":"9288ccb2-6e3c-4bfc-823d-a772e11cad90","Type":"ContainerDied","Data":"bfc6bab46604a5c8ac8937cda3491187e4e1a542317c96e62d7516dd77b4d7bf"}
Oct 30 22:11:25.385050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.384256    4581 generic.go:356] "Generic (PLEG): container finished" podID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerID="c1a7aaf8b80486f5783868807bbf2b20862b5cdd4af93ce458eab44004d897e2" exitCode=0
Oct 30 22:11:25.385050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.384351    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks" event={"ID":"8248e1e6-8e66-4f08-92e4-118f49638f02","Type":"ContainerDied","Data":"c1a7aaf8b80486f5783868807bbf2b20862b5cdd4af93ce458eab44004d897e2"}
Oct 30 22:11:25.391402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.390733    4581 generic.go:356] "Generic (PLEG): container finished" podID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerID="6d5bf131228617ce92685aa251e3890b9e60f15bae0c64b2164640d14d104cea" exitCode=0
Oct 30 22:11:25.391402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.390849    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz" event={"ID":"c1e6549a-b2b6-4156-b92d-64c5d8081ca5","Type":"ContainerDied","Data":"6d5bf131228617ce92685aa251e3890b9e60f15bae0c64b2164640d14d104cea"}
Oct 30 22:11:25.396815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.395738    4581 generic.go:356] "Generic (PLEG): container finished" podID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerID="cf23394bd984d27cfa38e199f0ada0c3bf9b1d5e6ef4de76c34221bda1c088fd" exitCode=0
Oct 30 22:11:25.396815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.395873    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk" event={"ID":"2e67d50a-4bf6-4045-8adf-50c1886284da","Type":"ContainerDied","Data":"cf23394bd984d27cfa38e199f0ada0c3bf9b1d5e6ef4de76c34221bda1c088fd"}
Oct 30 22:11:25.399730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.399701    4581 generic.go:356] "Generic (PLEG): container finished" podID="8a545860-1357-4871-b8d6-4a56f3c16183" containerID="6c94d24507058223bb708ee609b181e28878c6e88ec9eb258ac4aea384642c4d" exitCode=0
Oct 30 22:11:25.399846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.399812    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq" event={"ID":"8a545860-1357-4871-b8d6-4a56f3c16183","Type":"ContainerDied","Data":"6c94d24507058223bb708ee609b181e28878c6e88ec9eb258ac4aea384642c4d"}
Oct 30 22:11:25.403914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.403888    4581 generic.go:356] "Generic (PLEG): container finished" podID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerID="4dd51691844b4eb6748e961f864ddccc6469d022b55d23f365bd7db331e9ed88" exitCode=0
Oct 30 22:11:25.404197 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.404071    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28" event={"ID":"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4","Type":"ContainerDied","Data":"4dd51691844b4eb6748e961f864ddccc6469d022b55d23f365bd7db331e9ed88"}
Oct 30 22:11:25.411933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.411900    4581 generic.go:356] "Generic (PLEG): container finished" podID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerID="4edbdca3f8f6af33b5166642a17216d5c4d1054c7b85d0d59fdd128147ae4fba" exitCode=0
Oct 30 22:11:25.412060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:25.411945    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g" event={"ID":"9288ccb2-6e3c-4bfc-823d-a772e11cad90","Type":"ContainerDied","Data":"4edbdca3f8f6af33b5166642a17216d5c4d1054c7b85d0d59fdd128147ae4fba"}
Oct 30 22:11:26.427093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:26.426068    4581 generic.go:356] "Generic (PLEG): container finished" podID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerID="9e45ad324e771c10a503136e4a53ca4949b03abe96cd863a15366d8e6de5d792" exitCode=0
Oct 30 22:11:26.427093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:26.426124    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28" event={"ID":"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4","Type":"ContainerDied","Data":"9e45ad324e771c10a503136e4a53ca4949b03abe96cd863a15366d8e6de5d792"}
Oct 30 22:11:28.461703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:28.461661    4581 generic.go:356] "Generic (PLEG): container finished" podID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerID="145bea34cf8c528ebd8759346208af7b695e28841fe6f7c577950307b6648c48" exitCode=0
Oct 30 22:11:28.462155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:28.461715    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts" event={"ID":"f270a9c1-75f1-4d14-9e3b-058e0e019d41","Type":"ContainerDied","Data":"145bea34cf8c528ebd8759346208af7b695e28841fe6f7c577950307b6648c48"}
Oct 30 22:11:40.847234 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:11:40.847111    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:11:47.733020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:47.732377    4581 generic.go:356] "Generic (PLEG): container finished" podID="88ca6131-c91a-4488-923a-344f0c0d3338" containerID="d2710432f890664ad0f38e16e4dc170ccb3d6fa1b0754fb84cbf82c2b93e9cf2" exitCode=0
Oct 30 22:11:47.733020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:47.732869    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp" event={"ID":"88ca6131-c91a-4488-923a-344f0c0d3338","Type":"ContainerDied","Data":"d2710432f890664ad0f38e16e4dc170ccb3d6fa1b0754fb84cbf82c2b93e9cf2"}
Oct 30 22:11:47.741048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:47.740742    4581 generic.go:356] "Generic (PLEG): container finished" podID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerID="5aa0cfd7790b3db153a125b3ecf05b92aaca3da02cd0d37a8a630a1f69d0e47a" exitCode=0
Oct 30 22:11:47.741048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:47.740837    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj" event={"ID":"1bd86ea7-1705-4692-af20-2da63ff666a9","Type":"ContainerDied","Data":"5aa0cfd7790b3db153a125b3ecf05b92aaca3da02cd0d37a8a630a1f69d0e47a"}
Oct 30 22:11:47.745552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:47.744912    4581 generic.go:356] "Generic (PLEG): container finished" podID="14ef3be9-6595-4521-8aad-a8315a6add28" containerID="f3e322cf7199523c2b6071f862f6f9c09e19e510be35982ec6e96f9c93cd3f4e" exitCode=0
Oct 30 22:11:47.745552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:47.745061    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v" event={"ID":"14ef3be9-6595-4521-8aad-a8315a6add28","Type":"ContainerDied","Data":"f3e322cf7199523c2b6071f862f6f9c09e19e510be35982ec6e96f9c93cd3f4e"}
Oct 30 22:11:47.752331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:47.752294    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms" event={"ID":"a0e71a0d-794b-4960-b837-9f3331977139","Type":"ContainerStarted","Data":"f5fb50a5bf67ded26ca8873c8c75f6d4c14cee5ea9d9c919b7b294c5b2a4aa8e"}
Oct 30 22:11:47.816500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:47.814852    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms" podStartSLOduration=2.220710082 podStartE2EDuration="46.81483054s" totalImagesPullingTime="44.594120458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:47.814236583 +0000 UTC m=+291.853175460" watchObservedRunningTime="2026-10-30 22:11:47.81483054 +0000 UTC m=+291.853769427"
Oct 30 22:11:48.761882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.761844    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq" event={"ID":"8a545860-1357-4871-b8d6-4a56f3c16183","Type":"ContainerStarted","Data":"1702e7259f19501a4a62494853913ccccf66ea9b016ee232b6f478af9b5448f1"}
Oct 30 22:11:48.764882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.764849    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g" event={"ID":"9288ccb2-6e3c-4bfc-823d-a772e11cad90","Type":"ContainerStarted","Data":"5fa9f3d493b25dec9ce67e70894239985239e94c13e53b9cc0849c347b0620d3"}
Oct 30 22:11:48.767507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.767479    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz" event={"ID":"c1e6549a-b2b6-4156-b92d-64c5d8081ca5","Type":"ContainerStarted","Data":"7dd96f82c154ef03f308581dd824149e0d5e3460bbe234a53e7f7e33e7e1d7d8"}
Oct 30 22:11:48.770125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.769971    4581 generic.go:356] "Generic (PLEG): container finished" podID="a0e71a0d-794b-4960-b837-9f3331977139" containerID="f5fb50a5bf67ded26ca8873c8c75f6d4c14cee5ea9d9c919b7b294c5b2a4aa8e" exitCode=0
Oct 30 22:11:48.770125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.770082    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms" event={"ID":"a0e71a0d-794b-4960-b837-9f3331977139","Type":"ContainerDied","Data":"f5fb50a5bf67ded26ca8873c8c75f6d4c14cee5ea9d9c919b7b294c5b2a4aa8e"}
Oct 30 22:11:48.772994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.772961    4581 generic.go:356] "Generic (PLEG): container finished" podID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerID="d5370513fef0b15e337819860be8f8e546bd49841bd608edd8ef948075947d07" exitCode=0
Oct 30 22:11:48.773149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.773057    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk" event={"ID":"2e67d50a-4bf6-4045-8adf-50c1886284da","Type":"ContainerDied","Data":"d5370513fef0b15e337819860be8f8e546bd49841bd608edd8ef948075947d07"}
Oct 30 22:11:48.807456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.806886    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq" podStartSLOduration=2.669372521 podStartE2EDuration="48.806865814s" totalImagesPullingTime="46.137493293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:48.804833557 +0000 UTC m=+292.843772444" watchObservedRunningTime="2026-10-30 22:11:48.806865814 +0000 UTC m=+292.845804702"
Oct 30 22:11:48.896170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.896118    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz" podStartSLOduration=3.477844032 podStartE2EDuration="39.896103601s" totalImagesPullingTime="36.418259569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:48.89550954 +0000 UTC m=+292.934448429" watchObservedRunningTime="2026-10-30 22:11:48.896103601 +0000 UTC m=+292.935042486"
Oct 30 22:11:48.945094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:48.945033    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g" podStartSLOduration=3.484865897 podStartE2EDuration="39.945016947s" totalImagesPullingTime="36.46015105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:48.945013498 +0000 UTC m=+292.983952385" watchObservedRunningTime="2026-10-30 22:11:48.945016947 +0000 UTC m=+292.983955836"
Oct 30 22:11:49.011810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.011723    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:49.040449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.040418    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:49.063560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.063526    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:49.116566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.116533    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-util\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-util\") pod \"14ef3be9-6595-4521-8aad-a8315a6add28\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") "
Oct 30 22:11:49.116736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.116579    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ef3be9-6595-4521-8aad-a8315a6add28-kube-api-access-ck2sz\" (UniqueName: \"kubernetes.io/projected/14ef3be9-6595-4521-8aad-a8315a6add28-kube-api-access-ck2sz\") pod \"14ef3be9-6595-4521-8aad-a8315a6add28\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") "
Oct 30 22:11:49.116736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.116608    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-bundle\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-bundle\") pod \"14ef3be9-6595-4521-8aad-a8315a6add28\" (UID: \"14ef3be9-6595-4521-8aad-a8315a6add28\") "
Oct 30 22:11:49.117335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.117303    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-bundle" pod "14ef3be9-6595-4521-8aad-a8315a6add28" (UID: "14ef3be9-6595-4521-8aad-a8315a6add28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:49.119862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.119831    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ef3be9-6595-4521-8aad-a8315a6add28-kube-api-access-ck2sz" pod "14ef3be9-6595-4521-8aad-a8315a6add28" (UID: "14ef3be9-6595-4521-8aad-a8315a6add28"). InnerVolumeSpecName "kube-api-access-ck2sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:49.124613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.124582    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-util" pod "14ef3be9-6595-4521-8aad-a8315a6add28" (UID: "14ef3be9-6595-4521-8aad-a8315a6add28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:49.217433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217400    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd86ea7-1705-4692-af20-2da63ff666a9-kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/1bd86ea7-1705-4692-af20-2da63ff666a9-kube-api-access-r8tsm\") pod \"1bd86ea7-1705-4692-af20-2da63ff666a9\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") "
Oct 30 22:11:49.217582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217455    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-bundle\") pod \"1bd86ea7-1705-4692-af20-2da63ff666a9\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") "
Oct 30 22:11:49.217582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217502    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-util\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-util\") pod \"1bd86ea7-1705-4692-af20-2da63ff666a9\" (UID: \"1bd86ea7-1705-4692-af20-2da63ff666a9\") "
Oct 30 22:11:49.217582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217537    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-util\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-util\") pod \"88ca6131-c91a-4488-923a-344f0c0d3338\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") "
Oct 30 22:11:49.217582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217560    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-bundle\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-bundle\") pod \"88ca6131-c91a-4488-923a-344f0c0d3338\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") "
Oct 30 22:11:49.217726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217593    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ca6131-c91a-4488-923a-344f0c0d3338-kube-api-access-s52xv\" (UniqueName: \"kubernetes.io/projected/88ca6131-c91a-4488-923a-344f0c0d3338-kube-api-access-s52xv\") pod \"88ca6131-c91a-4488-923a-344f0c0d3338\" (UID: \"88ca6131-c91a-4488-923a-344f0c0d3338\") "
Oct 30 22:11:49.217896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217863    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.217896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217892    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck2sz\" (UniqueName: \"kubernetes.io/projected/14ef3be9-6595-4521-8aad-a8315a6add28-kube-api-access-ck2sz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.218041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.217907    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14ef3be9-6595-4521-8aad-a8315a6add28-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.218285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.218252    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-bundle" pod "88ca6131-c91a-4488-923a-344f0c0d3338" (UID: "88ca6131-c91a-4488-923a-344f0c0d3338"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:49.220025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.219987    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-bundle" pod "1bd86ea7-1705-4692-af20-2da63ff666a9" (UID: "1bd86ea7-1705-4692-af20-2da63ff666a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:49.220131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.220061    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd86ea7-1705-4692-af20-2da63ff666a9-kube-api-access-r8tsm" pod "1bd86ea7-1705-4692-af20-2da63ff666a9" (UID: "1bd86ea7-1705-4692-af20-2da63ff666a9"). InnerVolumeSpecName "kube-api-access-r8tsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:49.221269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.221244    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ca6131-c91a-4488-923a-344f0c0d3338-kube-api-access-s52xv" pod "88ca6131-c91a-4488-923a-344f0c0d3338" (UID: "88ca6131-c91a-4488-923a-344f0c0d3338"). InnerVolumeSpecName "kube-api-access-s52xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:49.224392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.224365    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-util" pod "1bd86ea7-1705-4692-af20-2da63ff666a9" (UID: "1bd86ea7-1705-4692-af20-2da63ff666a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:49.224953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.224929    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-util" pod "88ca6131-c91a-4488-923a-344f0c0d3338" (UID: "88ca6131-c91a-4488-923a-344f0c0d3338"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:49.319044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.319005    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s52xv\" (UniqueName: \"kubernetes.io/projected/88ca6131-c91a-4488-923a-344f0c0d3338-kube-api-access-s52xv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.319044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.319040    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/1bd86ea7-1705-4692-af20-2da63ff666a9-kube-api-access-r8tsm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.319044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.319050    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.319330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.319058    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd86ea7-1705-4692-af20-2da63ff666a9-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.319330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.319067    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.319330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.319074    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88ca6131-c91a-4488-923a-344f0c0d3338-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:49.779733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.779648    4581 generic.go:356] "Generic (PLEG): container finished" podID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerID="64fb30ae1d5626701a7b3f91d021b84b9a6d40e41fe7e7d9b6faac656b6fd8c1" exitCode=0
Oct 30 22:11:49.779733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.779722    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks" event={"ID":"8248e1e6-8e66-4f08-92e4-118f49638f02","Type":"ContainerDied","Data":"64fb30ae1d5626701a7b3f91d021b84b9a6d40e41fe7e7d9b6faac656b6fd8c1"}
Oct 30 22:11:49.781739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.781712    4581 generic.go:356] "Generic (PLEG): container finished" podID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerID="7dd96f82c154ef03f308581dd824149e0d5e3460bbe234a53e7f7e33e7e1d7d8" exitCode=0
Oct 30 22:11:49.781848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.781791    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz" event={"ID":"c1e6549a-b2b6-4156-b92d-64c5d8081ca5","Type":"ContainerDied","Data":"7dd96f82c154ef03f308581dd824149e0d5e3460bbe234a53e7f7e33e7e1d7d8"}
Oct 30 22:11:49.789004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.788975    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp" event={"ID":"88ca6131-c91a-4488-923a-344f0c0d3338","Type":"ContainerDied","Data":"4e2b90e3ccf4c53812b04889319070443951f029f332426cd12187641b6ebdc4"}
Oct 30 22:11:49.789004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.788987    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fm7qp"
Oct 30 22:11:49.789197 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.789056    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e2b90e3ccf4c53812b04889319070443951f029f332426cd12187641b6ebdc4"
Oct 30 22:11:49.791119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.791091    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj" event={"ID":"1bd86ea7-1705-4692-af20-2da63ff666a9","Type":"ContainerDied","Data":"810d07395508fa3e7310aeb87c3bf9fadd13adc6bc6d391ac222a35415b2f739"}
Oct 30 22:11:49.791259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.791122    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6hzj"
Oct 30 22:11:49.791259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.791141    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="810d07395508fa3e7310aeb87c3bf9fadd13adc6bc6d391ac222a35415b2f739"
Oct 30 22:11:49.793276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.793248    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v" event={"ID":"14ef3be9-6595-4521-8aad-a8315a6add28","Type":"ContainerDied","Data":"3e468ba65ac467b9ec62ff7fb3d558b5debf1754892f680774439e0eab26905f"}
Oct 30 22:11:49.793439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.793425    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e468ba65ac467b9ec62ff7fb3d558b5debf1754892f680774439e0eab26905f"
Oct 30 22:11:49.793525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.793258    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c879v"
Oct 30 22:11:49.795534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.795503    4581 generic.go:356] "Generic (PLEG): container finished" podID="8a545860-1357-4871-b8d6-4a56f3c16183" containerID="1702e7259f19501a4a62494853913ccccf66ea9b016ee232b6f478af9b5448f1" exitCode=0
Oct 30 22:11:49.795631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.795616    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq" event={"ID":"8a545860-1357-4871-b8d6-4a56f3c16183","Type":"ContainerDied","Data":"1702e7259f19501a4a62494853913ccccf66ea9b016ee232b6f478af9b5448f1"}
Oct 30 22:11:49.799107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.799083    4581 generic.go:356] "Generic (PLEG): container finished" podID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerID="664f9cef3fa258d63e730153d2a32c102bfb38e7fdeed4b4165f3877a9bc1299" exitCode=0
Oct 30 22:11:49.799200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.799161    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28" event={"ID":"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4","Type":"ContainerDied","Data":"664f9cef3fa258d63e730153d2a32c102bfb38e7fdeed4b4165f3877a9bc1299"}
Oct 30 22:11:49.801628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.801604    4581 generic.go:356] "Generic (PLEG): container finished" podID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerID="5fa9f3d493b25dec9ce67e70894239985239e94c13e53b9cc0849c347b0620d3" exitCode=0
Oct 30 22:11:49.801701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.801652    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g" event={"ID":"9288ccb2-6e3c-4bfc-823d-a772e11cad90","Type":"ContainerDied","Data":"5fa9f3d493b25dec9ce67e70894239985239e94c13e53b9cc0849c347b0620d3"}
Oct 30 22:11:49.940426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.940399    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:49.976338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:49.976313    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:50.125414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.125376    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e71a0d-794b-4960-b837-9f3331977139-kube-api-access-kq9sl\" (UniqueName: \"kubernetes.io/projected/a0e71a0d-794b-4960-b837-9f3331977139-kube-api-access-kq9sl\") pod \"a0e71a0d-794b-4960-b837-9f3331977139\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") "
Oct 30 22:11:50.125414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.125420    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-bundle\") pod \"2e67d50a-4bf6-4045-8adf-50c1886284da\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") "
Oct 30 22:11:50.125674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.125485    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-util\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-util\") pod \"a0e71a0d-794b-4960-b837-9f3331977139\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") "
Oct 30 22:11:50.125674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.125524    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-bundle\") pod \"a0e71a0d-794b-4960-b837-9f3331977139\" (UID: \"a0e71a0d-794b-4960-b837-9f3331977139\") "
Oct 30 22:11:50.125674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.125544    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-util\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-util\") pod \"2e67d50a-4bf6-4045-8adf-50c1886284da\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") "
Oct 30 22:11:50.125674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.125621    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e67d50a-4bf6-4045-8adf-50c1886284da-kube-api-access-gs5vs\" (UniqueName: \"kubernetes.io/projected/2e67d50a-4bf6-4045-8adf-50c1886284da-kube-api-access-gs5vs\") pod \"2e67d50a-4bf6-4045-8adf-50c1886284da\" (UID: \"2e67d50a-4bf6-4045-8adf-50c1886284da\") "
Oct 30 22:11:50.126072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.126043    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-bundle" pod "2e67d50a-4bf6-4045-8adf-50c1886284da" (UID: "2e67d50a-4bf6-4045-8adf-50c1886284da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:50.126335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.126305    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-bundle" pod "a0e71a0d-794b-4960-b837-9f3331977139" (UID: "a0e71a0d-794b-4960-b837-9f3331977139"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:50.127952 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.127912    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e71a0d-794b-4960-b837-9f3331977139-kube-api-access-kq9sl" pod "a0e71a0d-794b-4960-b837-9f3331977139" (UID: "a0e71a0d-794b-4960-b837-9f3331977139"). InnerVolumeSpecName "kube-api-access-kq9sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:50.128420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.128392    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e67d50a-4bf6-4045-8adf-50c1886284da-kube-api-access-gs5vs" pod "2e67d50a-4bf6-4045-8adf-50c1886284da" (UID: "2e67d50a-4bf6-4045-8adf-50c1886284da"). InnerVolumeSpecName "kube-api-access-gs5vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:50.132168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.132140    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-util" pod "2e67d50a-4bf6-4045-8adf-50c1886284da" (UID: "2e67d50a-4bf6-4045-8adf-50c1886284da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:50.132800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.132775    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-util" pod "a0e71a0d-794b-4960-b837-9f3331977139" (UID: "a0e71a0d-794b-4960-b837-9f3331977139"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:50.226255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.226190    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:50.226255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.226250    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0e71a0d-794b-4960-b837-9f3331977139-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:50.226255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.226259    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:50.226255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.226268    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs5vs\" (UniqueName: \"kubernetes.io/projected/2e67d50a-4bf6-4045-8adf-50c1886284da-kube-api-access-gs5vs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:50.226528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.226278    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq9sl\" (UniqueName: \"kubernetes.io/projected/a0e71a0d-794b-4960-b837-9f3331977139-kube-api-access-kq9sl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:50.226528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.226289    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e67d50a-4bf6-4045-8adf-50c1886284da-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:50.808444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.808411    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms" event={"ID":"a0e71a0d-794b-4960-b837-9f3331977139","Type":"ContainerDied","Data":"2a4f1ec8ea798938a1ebc2398cd96f79174564e882aea55c7e4e14bef9117494"}
Oct 30 22:11:50.808899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.808456    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4f1ec8ea798938a1ebc2398cd96f79174564e882aea55c7e4e14bef9117494"
Oct 30 22:11:50.808899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.808422    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amr5ms"
Oct 30 22:11:50.810441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.810419    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk"
Oct 30 22:11:50.810441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.810424    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hs9rk" event={"ID":"2e67d50a-4bf6-4045-8adf-50c1886284da","Type":"ContainerDied","Data":"45fe20b60cab11a213db165a346cb549bced5b8d70b86095e79bb7ff70eb9823"}
Oct 30 22:11:50.810638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.810468    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45fe20b60cab11a213db165a346cb549bced5b8d70b86095e79bb7ff70eb9823"
Oct 30 22:11:50.812562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.812536    4581 generic.go:356] "Generic (PLEG): container finished" podID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerID="e2d3e262004841dcdd351dcc9b9cc15ac88065f6f331d8af8d4cc0a21fa5bc1b" exitCode=0
Oct 30 22:11:50.812689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.812630    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts" event={"ID":"f270a9c1-75f1-4d14-9e3b-058e0e019d41","Type":"ContainerDied","Data":"e2d3e262004841dcdd351dcc9b9cc15ac88065f6f331d8af8d4cc0a21fa5bc1b"}
Oct 30 22:11:50.989421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:50.989390    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:51.034567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.034540    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:51.040815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.040787    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:51.050128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.050105    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:51.053975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.053949    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:51.132393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.132363    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-bundle\") pod \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") "
Oct 30 22:11:51.132585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.132438    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-kube-api-access-4rf4p\" (UniqueName: \"kubernetes.io/projected/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-kube-api-access-4rf4p\") pod \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") "
Oct 30 22:11:51.132585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.132479    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-util\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-util\") pod \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\" (UID: \"c1e6549a-b2b6-4156-b92d-64c5d8081ca5\") "
Oct 30 22:11:51.133140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.133065    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-bundle" pod "c1e6549a-b2b6-4156-b92d-64c5d8081ca5" (UID: "c1e6549a-b2b6-4156-b92d-64c5d8081ca5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.134955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.134927    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-kube-api-access-4rf4p" pod "c1e6549a-b2b6-4156-b92d-64c5d8081ca5" (UID: "c1e6549a-b2b6-4156-b92d-64c5d8081ca5"). InnerVolumeSpecName "kube-api-access-4rf4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:51.140357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.140323    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-util" pod "c1e6549a-b2b6-4156-b92d-64c5d8081ca5" (UID: "c1e6549a-b2b6-4156-b92d-64c5d8081ca5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.233235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233188    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8248e1e6-8e66-4f08-92e4-118f49638f02-kube-api-access-x4pfq\" (UniqueName: \"kubernetes.io/projected/8248e1e6-8e66-4f08-92e4-118f49638f02-kube-api-access-x4pfq\") pod \"8248e1e6-8e66-4f08-92e4-118f49638f02\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") "
Oct 30 22:11:51.233397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233265    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-util\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-util\") pod \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") "
Oct 30 22:11:51.233397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233285    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-bundle\") pod \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") "
Oct 30 22:11:51.233397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233304    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-kube-api-access-scndt\" (UniqueName: \"kubernetes.io/projected/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-kube-api-access-scndt\") pod \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") "
Oct 30 22:11:51.233397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233328    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9288ccb2-6e3c-4bfc-823d-a772e11cad90-kube-api-access-kpsvs\" (UniqueName: \"kubernetes.io/projected/9288ccb2-6e3c-4bfc-823d-a772e11cad90-kube-api-access-kpsvs\") pod \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") "
Oct 30 22:11:51.233397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233350    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-util\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-util\") pod \"8a545860-1357-4871-b8d6-4a56f3c16183\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") "
Oct 30 22:11:51.233397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233374    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-bundle\") pod \"8248e1e6-8e66-4f08-92e4-118f49638f02\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") "
Oct 30 22:11:51.233731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233710    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-bundle\") pod \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\" (UID: \"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4\") "
Oct 30 22:11:51.233935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233769    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-util\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-util\") pod \"8248e1e6-8e66-4f08-92e4-118f49638f02\" (UID: \"8248e1e6-8e66-4f08-92e4-118f49638f02\") "
Oct 30 22:11:51.234433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233875    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-bundle" pod "9288ccb2-6e3c-4bfc-823d-a772e11cad90" (UID: "9288ccb2-6e3c-4bfc-823d-a772e11cad90"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.234433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.234020    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a545860-1357-4871-b8d6-4a56f3c16183-kube-api-access-j6b7b\" (UniqueName: \"kubernetes.io/projected/8a545860-1357-4871-b8d6-4a56f3c16183-kube-api-access-j6b7b\") pod \"8a545860-1357-4871-b8d6-4a56f3c16183\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") "
Oct 30 22:11:51.234433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.233902    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-bundle" pod "8248e1e6-8e66-4f08-92e4-118f49638f02" (UID: "8248e1e6-8e66-4f08-92e4-118f49638f02"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.234899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.234884    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-bundle\") pod \"8a545860-1357-4871-b8d6-4a56f3c16183\" (UID: \"8a545860-1357-4871-b8d6-4a56f3c16183\") "
Oct 30 22:11:51.235012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.234998    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-util\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-util\") pod \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\" (UID: \"9288ccb2-6e3c-4bfc-823d-a772e11cad90\") "
Oct 30 22:11:51.235396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.235365    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.235491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.235480    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.235577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.235564    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rf4p\" (UniqueName: \"kubernetes.io/projected/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-kube-api-access-4rf4p\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.235663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.235643    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1e6549a-b2b6-4156-b92d-64c5d8081ca5-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.235663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.235664    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.235839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.235021    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-bundle" pod "5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" (UID: "5f297c2b-3ccd-4f6f-8da8-03f485b4eec4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.236251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.236210    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8248e1e6-8e66-4f08-92e4-118f49638f02-kube-api-access-x4pfq" pod "8248e1e6-8e66-4f08-92e4-118f49638f02" (UID: "8248e1e6-8e66-4f08-92e4-118f49638f02"). InnerVolumeSpecName "kube-api-access-x4pfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:51.236340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.236308    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-kube-api-access-scndt" pod "5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" (UID: "5f297c2b-3ccd-4f6f-8da8-03f485b4eec4"). InnerVolumeSpecName "kube-api-access-scndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:51.236401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.236384    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9288ccb2-6e3c-4bfc-823d-a772e11cad90-kube-api-access-kpsvs" pod "9288ccb2-6e3c-4bfc-823d-a772e11cad90" (UID: "9288ccb2-6e3c-4bfc-823d-a772e11cad90"). InnerVolumeSpecName "kube-api-access-kpsvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:51.236745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.236727    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-bundle" pod "8a545860-1357-4871-b8d6-4a56f3c16183" (UID: "8a545860-1357-4871-b8d6-4a56f3c16183"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.238118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.238101    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a545860-1357-4871-b8d6-4a56f3c16183-kube-api-access-j6b7b" pod "8a545860-1357-4871-b8d6-4a56f3c16183" (UID: "8a545860-1357-4871-b8d6-4a56f3c16183"). InnerVolumeSpecName "kube-api-access-j6b7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:51.242434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.242399    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-util" pod "5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" (UID: "5f297c2b-3ccd-4f6f-8da8-03f485b4eec4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.242542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.242477    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-util" pod "8a545860-1357-4871-b8d6-4a56f3c16183" (UID: "8a545860-1357-4871-b8d6-4a56f3c16183"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.243089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.243064    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-util" pod "8248e1e6-8e66-4f08-92e4-118f49638f02" (UID: "8248e1e6-8e66-4f08-92e4-118f49638f02"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.243878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.243860    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-util" pod "9288ccb2-6e3c-4bfc-823d-a772e11cad90" (UID: "9288ccb2-6e3c-4bfc-823d-a772e11cad90"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:51.336979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.336942    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scndt\" (UniqueName: \"kubernetes.io/projected/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-kube-api-access-scndt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.336979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.336974    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpsvs\" (UniqueName: \"kubernetes.io/projected/9288ccb2-6e3c-4bfc-823d-a772e11cad90-kube-api-access-kpsvs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.336979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.336985    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.337287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.336994    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.337287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.337002    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8248e1e6-8e66-4f08-92e4-118f49638f02-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.337287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.337010    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6b7b\" (UniqueName: \"kubernetes.io/projected/8a545860-1357-4871-b8d6-4a56f3c16183-kube-api-access-j6b7b\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.337287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.337018    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a545860-1357-4871-b8d6-4a56f3c16183-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.337287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.337027    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9288ccb2-6e3c-4bfc-823d-a772e11cad90-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.337287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.337035    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pfq\" (UniqueName: \"kubernetes.io/projected/8248e1e6-8e66-4f08-92e4-118f49638f02-kube-api-access-x4pfq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.337287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.337042    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f297c2b-3ccd-4f6f-8da8-03f485b4eec4-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:51.821576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.821120    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq"
Oct 30 22:11:51.821576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.821132    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq6bnq" event={"ID":"8a545860-1357-4871-b8d6-4a56f3c16183","Type":"ContainerDied","Data":"19b9c8c50789c851796f67a888f2c92fabc90115b2c58785c4896cedadb898d1"}
Oct 30 22:11:51.821576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.821237    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19b9c8c50789c851796f67a888f2c92fabc90115b2c58785c4896cedadb898d1"
Oct 30 22:11:51.825889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.825869    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28"
Oct 30 22:11:51.825889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.825874    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zbm28" event={"ID":"5f297c2b-3ccd-4f6f-8da8-03f485b4eec4","Type":"ContainerDied","Data":"71bc3040663b14945e76452d3d0508600b0e0676f5e049cdab014dd112b9a772"}
Oct 30 22:11:51.826074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.825927    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71bc3040663b14945e76452d3d0508600b0e0676f5e049cdab014dd112b9a772"
Oct 30 22:11:51.828315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.828286    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g" event={"ID":"9288ccb2-6e3c-4bfc-823d-a772e11cad90","Type":"ContainerDied","Data":"a5c61186e235b38baaf0bc2b4cd4d222cc279a748e9f63c5dec5aaa2eb4cb165"}
Oct 30 22:11:51.828446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.828297    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jb5g"
Oct 30 22:11:51.828513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.828442    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5c61186e235b38baaf0bc2b4cd4d222cc279a748e9f63c5dec5aaa2eb4cb165"
Oct 30 22:11:51.830788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.830759    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks" event={"ID":"8248e1e6-8e66-4f08-92e4-118f49638f02","Type":"ContainerDied","Data":"11051e4067e2fb7fbdc26f71657076f0324bfaa30793e46fa5651e012d70c726"}
Oct 30 22:11:51.830788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.830781    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwp8ks"
Oct 30 22:11:51.830956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.830807    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11051e4067e2fb7fbdc26f71657076f0324bfaa30793e46fa5651e012d70c726"
Oct 30 22:11:51.833209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.833186    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz"
Oct 30 22:11:51.833348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.833186    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af45fz" event={"ID":"c1e6549a-b2b6-4156-b92d-64c5d8081ca5","Type":"ContainerDied","Data":"ed2a4c970fd05fbcdcaaeb40fd4c009061a1a3e0ddaa240364f948df382c423e"}
Oct 30 22:11:51.833348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.833273    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed2a4c970fd05fbcdcaaeb40fd4c009061a1a3e0ddaa240364f948df382c423e"
Oct 30 22:11:51.961208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:51.961181    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:52.043685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.043651    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-util\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-util\") pod \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") "
Oct 30 22:11:52.043875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.043704    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-bundle\") pod \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") "
Oct 30 22:11:52.043875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.043725    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f270a9c1-75f1-4d14-9e3b-058e0e019d41-kube-api-access-n88ch\" (UniqueName: \"kubernetes.io/projected/f270a9c1-75f1-4d14-9e3b-058e0e019d41-kube-api-access-n88ch\") pod \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\" (UID: \"f270a9c1-75f1-4d14-9e3b-058e0e019d41\") "
Oct 30 22:11:52.044169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.044137    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-bundle" pod "f270a9c1-75f1-4d14-9e3b-058e0e019d41" (UID: "f270a9c1-75f1-4d14-9e3b-058e0e019d41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:52.046025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.045994    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f270a9c1-75f1-4d14-9e3b-058e0e019d41-kube-api-access-n88ch" pod "f270a9c1-75f1-4d14-9e3b-058e0e019d41" (UID: "f270a9c1-75f1-4d14-9e3b-058e0e019d41"). InnerVolumeSpecName "kube-api-access-n88ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:52.048588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.048562    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-util" pod "f270a9c1-75f1-4d14-9e3b-058e0e019d41" (UID: "f270a9c1-75f1-4d14-9e3b-058e0e019d41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:52.144665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.144624    4581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-util\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:52.144665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.144661    4581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f270a9c1-75f1-4d14-9e3b-058e0e019d41-bundle\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:52.144871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.144676    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n88ch\" (UniqueName: \"kubernetes.io/projected/f270a9c1-75f1-4d14-9e3b-058e0e019d41-kube-api-access-n88ch\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:11:52.840055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.840026    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts"
Oct 30 22:11:52.840055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.840043    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449v5ts" event={"ID":"f270a9c1-75f1-4d14-9e3b-058e0e019d41","Type":"ContainerDied","Data":"d6cea8f86cde4ed3dfe31a1a257b926490da6156e351e76c214feeb3d410be16"}
Oct 30 22:11:52.840612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:52.840087    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6cea8f86cde4ed3dfe31a1a257b926490da6156e351e76c214feeb3d410be16"
Oct 30 22:11:56.133454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.133426    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:11:56.133847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.133528    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:11:56.262432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262397    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"]
Oct 30 22:11:56.262676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262665    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerName="pull"
Oct 30 22:11:56.262716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262678    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerName="pull"
Oct 30 22:11:56.262716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262688    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerName="extract"
Oct 30 22:11:56.262716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262695    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerName="extract"
Oct 30 22:11:56.262716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262703    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerName="util"
Oct 30 22:11:56.262716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262708    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerName="util"
Oct 30 22:11:56.262716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262714    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerName="extract"
Oct 30 22:11:56.262716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262718    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262726    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262731    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262736    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a0e71a0d-794b-4960-b837-9f3331977139" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262741    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e71a0d-794b-4960-b837-9f3331977139" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262749    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262753    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262759    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8a545860-1357-4871-b8d6-4a56f3c16183" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262763    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a545860-1357-4871-b8d6-4a56f3c16183" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262769    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262773    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262778    4581 cpu_manager.go:414] "containerMap: removing container" podUID="88ca6131-c91a-4488-923a-344f0c0d3338" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262782    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ca6131-c91a-4488-923a-344f0c0d3338" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262787    4581 cpu_manager.go:414] "containerMap: removing container" podUID="14ef3be9-6595-4521-8aad-a8315a6add28" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262791    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ef3be9-6595-4521-8aad-a8315a6add28" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262796    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262800    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262809    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8a545860-1357-4871-b8d6-4a56f3c16183" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262813    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a545860-1357-4871-b8d6-4a56f3c16183" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262820    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262823    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262829    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262833    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262838    4581 cpu_manager.go:414] "containerMap: removing container" podUID="88ca6131-c91a-4488-923a-344f0c0d3338" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262841    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ca6131-c91a-4488-923a-344f0c0d3338" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262847    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a0e71a0d-794b-4960-b837-9f3331977139" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262852    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e71a0d-794b-4960-b837-9f3331977139" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262857    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262862    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262866    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a0e71a0d-794b-4960-b837-9f3331977139" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262870    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e71a0d-794b-4960-b837-9f3331977139" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262877    4581 cpu_manager.go:414] "containerMap: removing container" podUID="14ef3be9-6595-4521-8aad-a8315a6add28" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262881    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ef3be9-6595-4521-8aad-a8315a6add28" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262888    4581 cpu_manager.go:414] "containerMap: removing container" podUID="88ca6131-c91a-4488-923a-344f0c0d3338" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262893    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ca6131-c91a-4488-923a-344f0c0d3338" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262899    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262903    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262909    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262914    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerName="util"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262920    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262924    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerName="pull"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262930    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262934    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerName="extract"
Oct 30 22:11:56.262919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262940    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262944    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262951    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerName="pull"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262956    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerName="pull"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262964    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerName="pull"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262969    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerName="pull"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262977    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262982    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262990    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.262996    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263003    4581 cpu_manager.go:414] "containerMap: removing container" podUID="14ef3be9-6595-4521-8aad-a8315a6add28" containerName="util"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263007    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ef3be9-6595-4521-8aad-a8315a6add28" containerName="util"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263015    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8a545860-1357-4871-b8d6-4a56f3c16183" containerName="pull"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263018    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a545860-1357-4871-b8d6-4a56f3c16183" containerName="pull"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263024    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerName="pull"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263028    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerName="pull"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263071    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f297c2b-3ccd-4f6f-8da8-03f485b4eec4" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263080    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a545860-1357-4871-b8d6-4a56f3c16183" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263086    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ca6131-c91a-4488-923a-344f0c0d3338" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263091    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ef3be9-6595-4521-8aad-a8315a6add28" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263096    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9288ccb2-6e3c-4bfc-823d-a772e11cad90" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263102    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e71a0d-794b-4960-b837-9f3331977139" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263109    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f270a9c1-75f1-4d14-9e3b-058e0e019d41" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263113    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd86ea7-1705-4692-af20-2da63ff666a9" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263118    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e6549a-b2b6-4156-b92d-64c5d8081ca5" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263123    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8248e1e6-8e66-4f08-92e4-118f49638f02" containerName="extract"
Oct 30 22:11:56.264242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.263128    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e67d50a-4bf6-4045-8adf-50c1886284da" containerName="extract"
Oct 30 22:11:56.265552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.265534    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"
Oct 30 22:11:56.276933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.276905    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vhm\" (UniqueName: \"kubernetes.io/projected/ae9a5b6c-6320-49a3-821b-baf289ee8a24-kube-api-access-88vhm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz\" (UID: \"ae9a5b6c-6320-49a3-821b-baf289ee8a24\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"
Oct 30 22:11:56.277057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.276966    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9a5b6c-6320-49a3-821b-baf289ee8a24-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz\" (UID: \"ae9a5b6c-6320-49a3-821b-baf289ee8a24\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"
Oct 30 22:11:56.284442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.284420    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 22:11:56.293941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.293911    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:11:56.293941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.293933    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hrtlw\""
Oct 30 22:11:56.295814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.295790    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"]
Oct 30 22:11:56.378152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.378111    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vhm\" (UniqueName: \"kubernetes.io/projected/ae9a5b6c-6320-49a3-821b-baf289ee8a24-kube-api-access-88vhm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz\" (UID: \"ae9a5b6c-6320-49a3-821b-baf289ee8a24\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"
Oct 30 22:11:56.378390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.378176    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9a5b6c-6320-49a3-821b-baf289ee8a24-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz\" (UID: \"ae9a5b6c-6320-49a3-821b-baf289ee8a24\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"
Oct 30 22:11:56.378612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.378589    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9a5b6c-6320-49a3-821b-baf289ee8a24-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz\" (UID: \"ae9a5b6c-6320-49a3-821b-baf289ee8a24\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"
Oct 30 22:11:56.399508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.399434    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vhm\" (UniqueName: \"kubernetes.io/projected/ae9a5b6c-6320-49a3-821b-baf289ee8a24-kube-api-access-88vhm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz\" (UID: \"ae9a5b6c-6320-49a3-821b-baf289ee8a24\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"
Oct 30 22:11:56.578542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.578500    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"
Oct 30 22:11:56.900114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:56.899871    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz"]
Oct 30 22:11:56.904978 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:56.904948    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9a5b6c_6320_49a3_821b_baf289ee8a24.slice/crio-f2a7cc5ed5e640ac38a94695485659852927cd79000e42a3436f947896ace090 WatchSource:0}: Error finding container f2a7cc5ed5e640ac38a94695485659852927cd79000e42a3436f947896ace090: Status 404 returned error can't find the container with id f2a7cc5ed5e640ac38a94695485659852927cd79000e42a3436f947896ace090
Oct 30 22:11:57.654536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.654502    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"]
Oct 30 22:11:57.657476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.657453    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.674784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.674762    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 22:11:57.674960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.674793    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 22:11:57.676082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.676052    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 22:11:57.676201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.676052    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 22:11:57.676201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.676104    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 22:11:57.676345 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.676327    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-857vh\""
Oct 30 22:11:57.687763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.687734    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4562edf2-a964-4e20-a9fd-a687204831ff-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.687914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.687767    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4562edf2-a964-4e20-a9fd-a687204831ff-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.687914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.687801    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nnsp\" (UniqueName: \"kubernetes.io/projected/4562edf2-a964-4e20-a9fd-a687204831ff-kube-api-access-2nnsp\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.688041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.687912    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.688041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.687952    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.688041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.687988    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.789040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.788994    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4562edf2-a964-4e20-a9fd-a687204831ff-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.789040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.789040    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4562edf2-a964-4e20-a9fd-a687204831ff-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.789299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.789079    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nnsp\" (UniqueName: \"kubernetes.io/projected/4562edf2-a964-4e20-a9fd-a687204831ff-kube-api-access-2nnsp\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.789299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.789107    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.789299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.789130    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.789299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.789157    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.789552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.789451    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4562edf2-a964-4e20-a9fd-a687204831ff-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.791869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.791837    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4562edf2-a964-4e20-a9fd-a687204831ff-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.792562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.792537    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.792901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.792877    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.793083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.793057    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4562edf2-a964-4e20-a9fd-a687204831ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.813992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.813955    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nnsp\" (UniqueName: \"kubernetes.io/projected/4562edf2-a964-4e20-a9fd-a687204831ff-kube-api-access-2nnsp\") pod \"opendatahub-operator-controller-manager-c4c4484-xhqqd\" (UID: \"4562edf2-a964-4e20-a9fd-a687204831ff\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:57.836560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.836523    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"]
Oct 30 22:11:57.865951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.865899    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz" event={"ID":"ae9a5b6c-6320-49a3-821b-baf289ee8a24","Type":"ContainerStarted","Data":"f2a7cc5ed5e640ac38a94695485659852927cd79000e42a3436f947896ace090"}
Oct 30 22:11:57.974409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:57.974312    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:11:58.338143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:58.338113    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"]
Oct 30 22:11:58.340268 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:11:58.340241    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4562edf2_a964_4e20_a9fd_a687204831ff.slice/crio-4112e21d09fed0e4cfd37b298a36fdea8996ef53845ccf843335359a1279fffb WatchSource:0}: Error finding container 4112e21d09fed0e4cfd37b298a36fdea8996ef53845ccf843335359a1279fffb: Status 404 returned error can't find the container with id 4112e21d09fed0e4cfd37b298a36fdea8996ef53845ccf843335359a1279fffb
Oct 30 22:11:58.342334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:58.342311    4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:11:58.876300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:58.876263    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz" event={"ID":"ae9a5b6c-6320-49a3-821b-baf289ee8a24","Type":"ContainerStarted","Data":"d394520f5f6e037b0ebd2785a15fe4f91e307948a27231b664fe77dc23611ac4"}
Oct 30 22:11:58.878061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:58.877999    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd" event={"ID":"4562edf2-a964-4e20-a9fd-a687204831ff","Type":"ContainerStarted","Data":"4112e21d09fed0e4cfd37b298a36fdea8996ef53845ccf843335359a1279fffb"}
Oct 30 22:11:58.919080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:11:58.918359    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g7kqz" podStartSLOduration=1.5657023620000001 podStartE2EDuration="2.918338102s" totalImagesPullingTime="1.35263574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:58.916783059 +0000 UTC m=+302.955721947" watchObservedRunningTime="2026-10-30 22:11:58.918338102 +0000 UTC m=+302.957276989"
Oct 30 22:12:01.893891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:01.893856    4581 generic.go:356] "Generic (PLEG): container finished" podID="4562edf2-a964-4e20-a9fd-a687204831ff" containerID="d1abba377b9f0695e3f6f0a4688a45ee75e447720a2da6261f23f07d143dd91c" exitCode=0
Oct 30 22:12:01.894404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:01.893910    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd" event={"ID":"4562edf2-a964-4e20-a9fd-a687204831ff","Type":"ContainerDied","Data":"d1abba377b9f0695e3f6f0a4688a45ee75e447720a2da6261f23f07d143dd91c"}
Oct 30 22:12:02.856254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.856204    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"]
Oct 30 22:12:02.858738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.858718    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:02.882739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.882674    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"]
Oct 30 22:12:02.901830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.901788    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd" event={"ID":"4562edf2-a964-4e20-a9fd-a687204831ff","Type":"ContainerStarted","Data":"7b4c318ad9c37e607231a2746b37a39a11ac55026beeb6e0f982f943b3f36c5b"}
Oct 30 22:12:02.902365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.901914    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:12:02.934121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.934087    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 22:12:02.934121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.934116    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 22:12:02.934358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.934190    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fwxxw\""
Oct 30 22:12:02.944399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.944372    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7e5b659-2162-4552-919d-40c9866fd577-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-29lrm\" (UID: \"b7e5b659-2162-4552-919d-40c9866fd577\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:02.944562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.944518    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5d5\" (UniqueName: \"kubernetes.io/projected/b7e5b659-2162-4552-919d-40c9866fd577-kube-api-access-kb5d5\") pod \"cert-manager-webhook-5dbbbf75cc-29lrm\" (UID: \"b7e5b659-2162-4552-919d-40c9866fd577\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:02.970177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:02.970110    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd" podStartSLOduration=2.467452937 podStartE2EDuration="5.970095766s" totalImagesPullingTime="3.063778645s" totalInitContainerRuntime="438.864184ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:02.958662829 +0000 UTC m=+306.997601717" watchObservedRunningTime="2026-10-30 22:12:02.970095766 +0000 UTC m=+307.009034728"
Oct 30 22:12:03.045901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:03.045862    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5d5\" (UniqueName: \"kubernetes.io/projected/b7e5b659-2162-4552-919d-40c9866fd577-kube-api-access-kb5d5\") pod \"cert-manager-webhook-5dbbbf75cc-29lrm\" (UID: \"b7e5b659-2162-4552-919d-40c9866fd577\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:03.046061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:03.045928    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7e5b659-2162-4552-919d-40c9866fd577-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-29lrm\" (UID: \"b7e5b659-2162-4552-919d-40c9866fd577\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:03.063606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:03.063568    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5d5\" (UniqueName: \"kubernetes.io/projected/b7e5b659-2162-4552-919d-40c9866fd577-kube-api-access-kb5d5\") pod \"cert-manager-webhook-5dbbbf75cc-29lrm\" (UID: \"b7e5b659-2162-4552-919d-40c9866fd577\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:03.063606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:03.063580    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7e5b659-2162-4552-919d-40c9866fd577-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-29lrm\" (UID: \"b7e5b659-2162-4552-919d-40c9866fd577\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:03.178095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:03.178010    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:03.484667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:03.484582    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"]
Oct 30 22:12:03.488559 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:12:03.488523    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7e5b659_2162_4552_919d_40c9866fd577.slice/crio-39a65fc299a1f66b32c4815faaea87581848a4508454b2601eb0c346f5893c88 WatchSource:0}: Error finding container 39a65fc299a1f66b32c4815faaea87581848a4508454b2601eb0c346f5893c88: Status 404 returned error can't find the container with id 39a65fc299a1f66b32c4815faaea87581848a4508454b2601eb0c346f5893c88
Oct 30 22:12:03.908251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:03.908198    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" event={"ID":"b7e5b659-2162-4552-919d-40c9866fd577","Type":"ContainerStarted","Data":"39a65fc299a1f66b32c4815faaea87581848a4508454b2601eb0c346f5893c88"}
Oct 30 22:12:06.929611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:06.929569    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" event={"ID":"b7e5b659-2162-4552-919d-40c9866fd577","Type":"ContainerStarted","Data":"922e91ecdfe899491390ab44ada595c793d01bade13e7dfb01c638e78df430fb"}
Oct 30 22:12:06.930015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:06.929645    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:06.966337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:06.966271    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" podStartSLOduration=1.925102077 podStartE2EDuration="4.966252578s" totalImagesPullingTime="3.041150501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:06.964316869 +0000 UTC m=+311.003255755" watchObservedRunningTime="2026-10-30 22:12:06.966252578 +0000 UTC m=+311.005191465"
Oct 30 22:12:10.816883 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:12:10.816834    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:12:10.818291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:10.818260    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-5tczz"]
Oct 30 22:12:10.849020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:10.848979    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-5tczz"]
Oct 30 22:12:10.849180 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:10.849083    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-5tczz"
Oct 30 22:12:10.870835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:10.870809    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vgtrl\""
Oct 30 22:12:10.912595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:10.912555    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0324afe5-7060-4fc0-a8d1-0b92a896a490-bound-sa-token\") pod \"cert-manager-597b95cb84-5tczz\" (UID: \"0324afe5-7060-4fc0-a8d1-0b92a896a490\") " pod="cert-manager/cert-manager-597b95cb84-5tczz"
Oct 30 22:12:10.912768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:10.912639    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9cc5\" (UniqueName: \"kubernetes.io/projected/0324afe5-7060-4fc0-a8d1-0b92a896a490-kube-api-access-r9cc5\") pod \"cert-manager-597b95cb84-5tczz\" (UID: \"0324afe5-7060-4fc0-a8d1-0b92a896a490\") " pod="cert-manager/cert-manager-597b95cb84-5tczz"
Oct 30 22:12:11.013738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.013698    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9cc5\" (UniqueName: \"kubernetes.io/projected/0324afe5-7060-4fc0-a8d1-0b92a896a490-kube-api-access-r9cc5\") pod \"cert-manager-597b95cb84-5tczz\" (UID: \"0324afe5-7060-4fc0-a8d1-0b92a896a490\") " pod="cert-manager/cert-manager-597b95cb84-5tczz"
Oct 30 22:12:11.013738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.013748    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0324afe5-7060-4fc0-a8d1-0b92a896a490-bound-sa-token\") pod \"cert-manager-597b95cb84-5tczz\" (UID: \"0324afe5-7060-4fc0-a8d1-0b92a896a490\") " pod="cert-manager/cert-manager-597b95cb84-5tczz"
Oct 30 22:12:11.031430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.031398    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9cc5\" (UniqueName: \"kubernetes.io/projected/0324afe5-7060-4fc0-a8d1-0b92a896a490-kube-api-access-r9cc5\") pod \"cert-manager-597b95cb84-5tczz\" (UID: \"0324afe5-7060-4fc0-a8d1-0b92a896a490\") " pod="cert-manager/cert-manager-597b95cb84-5tczz"
Oct 30 22:12:11.031573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.031509    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0324afe5-7060-4fc0-a8d1-0b92a896a490-bound-sa-token\") pod \"cert-manager-597b95cb84-5tczz\" (UID: \"0324afe5-7060-4fc0-a8d1-0b92a896a490\") " pod="cert-manager/cert-manager-597b95cb84-5tczz"
Oct 30 22:12:11.180708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.180676    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-5tczz"
Oct 30 22:12:11.479042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.478957    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-5tczz"]
Oct 30 22:12:11.483211 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:12:11.483186    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0324afe5_7060_4fc0_a8d1_0b92a896a490.slice/crio-74e106c32780dc74c094eef55b6015ffb269cfad006589004c3c7e3d9fa1a47e WatchSource:0}: Error finding container 74e106c32780dc74c094eef55b6015ffb269cfad006589004c3c7e3d9fa1a47e: Status 404 returned error can't find the container with id 74e106c32780dc74c094eef55b6015ffb269cfad006589004c3c7e3d9fa1a47e
Oct 30 22:12:11.955591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.955552    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-5tczz" event={"ID":"0324afe5-7060-4fc0-a8d1-0b92a896a490","Type":"ContainerStarted","Data":"d480183e7621c65395ce86bae9826a6751c6ccc7e13e243c2978fc80714a0779"}
Oct 30 22:12:11.955591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.955598    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-5tczz" event={"ID":"0324afe5-7060-4fc0-a8d1-0b92a896a490","Type":"ContainerStarted","Data":"74e106c32780dc74c094eef55b6015ffb269cfad006589004c3c7e3d9fa1a47e"}
Oct 30 22:12:11.990373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:11.990324    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-5tczz" podStartSLOduration=1.9903100230000002 podStartE2EDuration="1.990310023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:11.990010856 +0000 UTC m=+316.028949743" watchObservedRunningTime="2026-10-30 22:12:11.990310023 +0000 UTC m=+316.029248910"
Oct 30 22:12:12.719104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.719069    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"]
Oct 30 22:12:12.721561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.721538    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"
Oct 30 22:12:12.734662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.734644    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 22:12:12.744151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.744136    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 22:12:12.749636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.749618    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7k48c\""
Oct 30 22:12:12.751451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.751431    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"]
Oct 30 22:12:12.828568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.828533    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x2f2\" (UniqueName: \"kubernetes.io/projected/8b0cc2fa-0371-42f4-8f82-9392f6d19485-kube-api-access-5x2f2\") pod \"authorino-operator-7b7bb4bc77-hkp44\" (UID: \"8b0cc2fa-0371-42f4-8f82-9392f6d19485\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"
Oct 30 22:12:12.929783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.929737    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2f2\" (UniqueName: \"kubernetes.io/projected/8b0cc2fa-0371-42f4-8f82-9392f6d19485-kube-api-access-5x2f2\") pod \"authorino-operator-7b7bb4bc77-hkp44\" (UID: \"8b0cc2fa-0371-42f4-8f82-9392f6d19485\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"
Oct 30 22:12:12.937644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.937617    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm"
Oct 30 22:12:12.952487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:12.952462    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x2f2\" (UniqueName: \"kubernetes.io/projected/8b0cc2fa-0371-42f4-8f82-9392f6d19485-kube-api-access-5x2f2\") pod \"authorino-operator-7b7bb4bc77-hkp44\" (UID: \"8b0cc2fa-0371-42f4-8f82-9392f6d19485\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"
Oct 30 22:12:13.033417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:13.033331    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"
Oct 30 22:12:13.336679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:13.336563    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"]
Oct 30 22:12:13.340893 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:12:13.340853    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0cc2fa_0371_42f4_8f82_9392f6d19485.slice/crio-e1066fe843508023025c3c22e8681cf957bcfbe60b6c523384b7c3f986859f3c WatchSource:0}: Error finding container e1066fe843508023025c3c22e8681cf957bcfbe60b6c523384b7c3f986859f3c: Status 404 returned error can't find the container with id e1066fe843508023025c3c22e8681cf957bcfbe60b6c523384b7c3f986859f3c
Oct 30 22:12:13.911107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:13.911072    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-xhqqd"
Oct 30 22:12:13.968677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:13.968632    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44" event={"ID":"8b0cc2fa-0371-42f4-8f82-9392f6d19485","Type":"ContainerStarted","Data":"e1066fe843508023025c3c22e8681cf957bcfbe60b6c523384b7c3f986859f3c"}
Oct 30 22:12:16.057566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.057534    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"]
Oct 30 22:12:16.060864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.060836    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"
Oct 30 22:12:16.077759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.077732    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-449l4\""
Oct 30 22:12:16.095182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.095148    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"]
Oct 30 22:12:16.162476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.162403    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjt8b\" (UniqueName: \"kubernetes.io/projected/a8227528-1913-4c13-ae2f-fa15e9e27876-kube-api-access-zjt8b\") pod \"limitador-operator-controller-manager-5778bb4449-9nhzz\" (UID: \"a8227528-1913-4c13-ae2f-fa15e9e27876\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"
Oct 30 22:12:16.263760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.263675    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjt8b\" (UniqueName: \"kubernetes.io/projected/a8227528-1913-4c13-ae2f-fa15e9e27876-kube-api-access-zjt8b\") pod \"limitador-operator-controller-manager-5778bb4449-9nhzz\" (UID: \"a8227528-1913-4c13-ae2f-fa15e9e27876\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"
Oct 30 22:12:16.286676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.286639    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjt8b\" (UniqueName: \"kubernetes.io/projected/a8227528-1913-4c13-ae2f-fa15e9e27876-kube-api-access-zjt8b\") pod \"limitador-operator-controller-manager-5778bb4449-9nhzz\" (UID: \"a8227528-1913-4c13-ae2f-fa15e9e27876\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"
Oct 30 22:12:16.373236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.373178    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"
Oct 30 22:12:16.673181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.673146    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"]
Oct 30 22:12:16.677231 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:12:16.677185    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8227528_1913_4c13_ae2f_fa15e9e27876.slice/crio-6607477b568e696c0d4b69822c58b00ce074a125232c614729be748259d41b6b WatchSource:0}: Error finding container 6607477b568e696c0d4b69822c58b00ce074a125232c614729be748259d41b6b: Status 404 returned error can't find the container with id 6607477b568e696c0d4b69822c58b00ce074a125232c614729be748259d41b6b
Oct 30 22:12:16.986308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.986191    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44" event={"ID":"8b0cc2fa-0371-42f4-8f82-9392f6d19485","Type":"ContainerStarted","Data":"de25ea33b7b9fe5d277638e784e84c84733746ddf50e8e58d3163aabf8db61ce"}
Oct 30 22:12:16.986308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.986275    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"
Oct 30 22:12:16.987622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:16.987599    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz" event={"ID":"a8227528-1913-4c13-ae2f-fa15e9e27876","Type":"ContainerStarted","Data":"6607477b568e696c0d4b69822c58b00ce074a125232c614729be748259d41b6b"}
Oct 30 22:12:17.026306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:17.026245    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44" podStartSLOduration=2.401060014 podStartE2EDuration="5.026209408s" totalImagesPullingTime="2.625149394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:17.023825753 +0000 UTC m=+321.062764638" watchObservedRunningTime="2026-10-30 22:12:17.026209408 +0000 UTC m=+321.065148358"
Oct 30 22:12:18.997863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:18.997831    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz" event={"ID":"a8227528-1913-4c13-ae2f-fa15e9e27876","Type":"ContainerStarted","Data":"d1fa508e1b43677e53cf9e815be03b9b481cced1ce5b89f2ae7dd611009d5a29"}
Oct 30 22:12:18.998306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:18.997961    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"
Oct 30 22:12:19.036629 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:19.036565    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz" podStartSLOduration=1.7389400529999999 podStartE2EDuration="3.036546749s" totalImagesPullingTime="1.297606696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:19.033991566 +0000 UTC m=+323.072930452" watchObservedRunningTime="2026-10-30 22:12:19.036546749 +0000 UTC m=+323.075485636"
Oct 30 22:12:22.194311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.194274    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"]
Oct 30 22:12:22.196750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.196733    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:22.210888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.210866    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bwgbh\""
Oct 30 22:12:22.223374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.223340    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"]
Oct 30 22:12:22.319841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.319799    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gln7\" (UniqueName: \"kubernetes.io/projected/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-kube-api-access-4gln7\") pod \"kuadrant-operator-controller-manager-678d87766f-hnqcq\" (UID: \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:22.320017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.319910    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hnqcq\" (UID: \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:22.420594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.420557    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hnqcq\" (UID: \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:22.420769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.420618    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gln7\" (UniqueName: \"kubernetes.io/projected/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-kube-api-access-4gln7\") pod \"kuadrant-operator-controller-manager-678d87766f-hnqcq\" (UID: \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:22.420994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.420973    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hnqcq\" (UID: \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:22.439135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.439107    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gln7\" (UniqueName: \"kubernetes.io/projected/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-kube-api-access-4gln7\") pod \"kuadrant-operator-controller-manager-678d87766f-hnqcq\" (UID: \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:22.508261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.508144    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:22.877933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:22.877902    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"]
Oct 30 22:12:22.881485 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:12:22.881449    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1f1049_4f70_4093_8cb8_8081e40c1e1e.slice/crio-1e98b7fce969078415cd55f14db87219c255832d999bfc7c4568eec904dadba2 WatchSource:0}: Error finding container 1e98b7fce969078415cd55f14db87219c255832d999bfc7c4568eec904dadba2: Status 404 returned error can't find the container with id 1e98b7fce969078415cd55f14db87219c255832d999bfc7c4568eec904dadba2
Oct 30 22:12:23.017879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:23.017847    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq" event={"ID":"1b1f1049-4f70-4093-8cb8-8081e40c1e1e","Type":"ContainerStarted","Data":"1e98b7fce969078415cd55f14db87219c255832d999bfc7c4568eec904dadba2"}
Oct 30 22:12:27.044704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:27.044663    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq" event={"ID":"1b1f1049-4f70-4093-8cb8-8081e40c1e1e","Type":"ContainerStarted","Data":"a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87"}
Oct 30 22:12:27.045116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:27.044751    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:27.117405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:27.117339    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq" podStartSLOduration=1.4975053059999999 podStartE2EDuration="5.1173244s" totalImagesPullingTime="3.619819094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:27.105615531 +0000 UTC m=+331.144554416" watchObservedRunningTime="2026-10-30 22:12:27.1173244 +0000 UTC m=+331.156263287"
Oct 30 22:12:27.994901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:27.994868    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hkp44"
Oct 30 22:12:28.424644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.424608    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"]
Oct 30 22:12:28.428754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.428728    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.460967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.460937    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"]
Oct 30 22:12:28.484503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.484465    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-z6vpf\""
Oct 30 22:12:28.578835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.578794    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26534606-c9fc-4de9-a421-8c49e98daeef-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.578835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.578840    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26534606-c9fc-4de9-a421-8c49e98daeef-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.579050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.578912    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.579050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.578934    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.579050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.578983    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26534606-c9fc-4de9-a421-8c49e98daeef-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.579148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.579074    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.579148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.579117    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.579208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.579143    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrpx\" (UniqueName: \"kubernetes.io/projected/26534606-c9fc-4de9-a421-8c49e98daeef-kube-api-access-kcrpx\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.579208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.579199    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.680526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680430    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26534606-c9fc-4de9-a421-8c49e98daeef-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.680526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680478    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.680526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680515    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.680800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680542    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrpx\" (UniqueName: \"kubernetes.io/projected/26534606-c9fc-4de9-a421-8c49e98daeef-kube-api-access-kcrpx\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.680800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680581    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.680800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680624    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26534606-c9fc-4de9-a421-8c49e98daeef-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.680800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680655    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26534606-c9fc-4de9-a421-8c49e98daeef-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.680800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680683    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.681048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680876    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.681048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680951    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.681048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.680987    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.681235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.681194    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.681329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.681305    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.681656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.681633    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26534606-c9fc-4de9-a421-8c49e98daeef-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.683206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.683182    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26534606-c9fc-4de9-a421-8c49e98daeef-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.683327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.683258    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26534606-c9fc-4de9-a421-8c49e98daeef-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.705032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.705000    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrpx\" (UniqueName: \"kubernetes.io/projected/26534606-c9fc-4de9-a421-8c49e98daeef-kube-api-access-kcrpx\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.706081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.706055    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26534606-c9fc-4de9-a421-8c49e98daeef-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm\" (UID: \"26534606-c9fc-4de9-a421-8c49e98daeef\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:28.742826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:28.742784    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:29.068817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:29.068744    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"]
Oct 30 22:12:29.071921 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:12:29.071889    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26534606_c9fc_4de9_a421_8c49e98daeef.slice/crio-d0110d7514783a4991124be8043aed29ca6e0ad9756b4e53991695adbce8f8fc WatchSource:0}: Error finding container d0110d7514783a4991124be8043aed29ca6e0ad9756b4e53991695adbce8f8fc: Status 404 returned error can't find the container with id d0110d7514783a4991124be8043aed29ca6e0ad9756b4e53991695adbce8f8fc
Oct 30 22:12:29.074176 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:29.074146    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:29.074256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:29.074211    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:29.074291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:29.074271    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:12:30.005242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:30.005173    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9nhzz"
Oct 30 22:12:30.063627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:30.063587    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm" event={"ID":"26534606-c9fc-4de9-a421-8c49e98daeef","Type":"ContainerStarted","Data":"fbe1a15b680d6bb75fbe8d6ef87d5f2301b194538a3299b7e352e37c3c25e8a9"}
Oct 30 22:12:30.063812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:30.063652    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm" event={"ID":"26534606-c9fc-4de9-a421-8c49e98daeef","Type":"ContainerStarted","Data":"d0110d7514783a4991124be8043aed29ca6e0ad9756b4e53991695adbce8f8fc"}
Oct 30 22:12:30.102243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:30.102128    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm" podStartSLOduration=2.102111926 podStartE2EDuration="2.102111926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:30.099948174 +0000 UTC m=+334.138887060" watchObservedRunningTime="2026-10-30 22:12:30.102111926 +0000 UTC m=+334.141050812"
Oct 30 22:12:30.743247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:30.743182    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:30.748108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:30.748080    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:31.069050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.068975    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:31.069996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.069980    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm"
Oct 30 22:12:31.204611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.204576    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4trf5"]
Oct 30 22:12:31.208923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.208898    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4trf5"
Oct 30 22:12:31.224630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.224594    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xwk44\""
Oct 30 22:12:31.226635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.226608    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4trf5"]
Oct 30 22:12:31.307955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.307910    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s79x6\" (UniqueName: \"kubernetes.io/projected/ee472ef0-20a0-43e4-9b0a-760f836ad083-kube-api-access-s79x6\") pod \"authorino-57f7f8659f-4trf5\" (UID: \"ee472ef0-20a0-43e4-9b0a-760f836ad083\") " pod="kuadrant-system/authorino-57f7f8659f-4trf5"
Oct 30 22:12:31.409307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.409275    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s79x6\" (UniqueName: \"kubernetes.io/projected/ee472ef0-20a0-43e4-9b0a-760f836ad083-kube-api-access-s79x6\") pod \"authorino-57f7f8659f-4trf5\" (UID: \"ee472ef0-20a0-43e4-9b0a-760f836ad083\") " pod="kuadrant-system/authorino-57f7f8659f-4trf5"
Oct 30 22:12:31.426540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.426513    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s79x6\" (UniqueName: \"kubernetes.io/projected/ee472ef0-20a0-43e4-9b0a-760f836ad083-kube-api-access-s79x6\") pod \"authorino-57f7f8659f-4trf5\" (UID: \"ee472ef0-20a0-43e4-9b0a-760f836ad083\") " pod="kuadrant-system/authorino-57f7f8659f-4trf5"
Oct 30 22:12:31.522678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.522640    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4trf5"
Oct 30 22:12:31.822814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:31.822736    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4trf5"]
Oct 30 22:12:31.826743 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:12:31.826714    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee472ef0_20a0_43e4_9b0a_760f836ad083.slice/crio-57d0064d6279b967bbb2020cab1240b8afadc3c16e46817f212322a58fcad826 WatchSource:0}: Error finding container 57d0064d6279b967bbb2020cab1240b8afadc3c16e46817f212322a58fcad826: Status 404 returned error can't find the container with id 57d0064d6279b967bbb2020cab1240b8afadc3c16e46817f212322a58fcad826
Oct 30 22:12:32.075689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:32.075596    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4trf5" event={"ID":"ee472ef0-20a0-43e4-9b0a-760f836ad083","Type":"ContainerStarted","Data":"57d0064d6279b967bbb2020cab1240b8afadc3c16e46817f212322a58fcad826"}
Oct 30 22:12:34.093521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:34.093489    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4trf5" event={"ID":"ee472ef0-20a0-43e4-9b0a-760f836ad083","Type":"ContainerStarted","Data":"0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0"}
Oct 30 22:12:34.121523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:34.121471    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4trf5" podStartSLOduration=1.363933357 podStartE2EDuration="3.121457478s" totalImagesPullingTime="1.757524121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:34.12130694 +0000 UTC m=+338.160245825" watchObservedRunningTime="2026-10-30 22:12:34.121457478 +0000 UTC m=+338.160396365"
Oct 30 22:12:34.713359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:34.713326    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4trf5"]
Oct 30 22:12:36.099289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:36.099206    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4trf5" podUID="ee472ef0-20a0-43e4-9b0a-760f836ad083" containerName="authorino" containerID="cri-o://0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0" gracePeriod=30
Oct 30 22:12:36.353623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:36.353602    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4trf5"
Oct 30 22:12:36.459312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:36.459266    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee472ef0-20a0-43e4-9b0a-760f836ad083-kube-api-access-s79x6\" (UniqueName: \"kubernetes.io/projected/ee472ef0-20a0-43e4-9b0a-760f836ad083-kube-api-access-s79x6\") pod \"ee472ef0-20a0-43e4-9b0a-760f836ad083\" (UID: \"ee472ef0-20a0-43e4-9b0a-760f836ad083\") "
Oct 30 22:12:36.461762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:36.461733    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee472ef0-20a0-43e4-9b0a-760f836ad083-kube-api-access-s79x6" pod "ee472ef0-20a0-43e4-9b0a-760f836ad083" (UID: "ee472ef0-20a0-43e4-9b0a-760f836ad083"). InnerVolumeSpecName "kube-api-access-s79x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:36.559861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:36.559828    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s79x6\" (UniqueName: \"kubernetes.io/projected/ee472ef0-20a0-43e4-9b0a-760f836ad083-kube-api-access-s79x6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:12:37.034757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.034717    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"]
Oct 30 22:12:37.035380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.035355    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ee472ef0-20a0-43e4-9b0a-760f836ad083" containerName="authorino"
Oct 30 22:12:37.035380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.035376    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee472ef0-20a0-43e4-9b0a-760f836ad083" containerName="authorino"
Oct 30 22:12:37.035559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.035454    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee472ef0-20a0-43e4-9b0a-760f836ad083" containerName="authorino"
Oct 30 22:12:37.038640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.038622    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.050830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.050803    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:12:37.050830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.050820    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 22:12:37.060042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.060019    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 22:12:37.060246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.060202    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 22:12:37.060325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.060283    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-c9kkr\""
Oct 30 22:12:37.060431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.060413    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 22:12:37.063340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.063320    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"]
Oct 30 22:12:37.105710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.105679    4581 generic.go:356] "Generic (PLEG): container finished" podID="ee472ef0-20a0-43e4-9b0a-760f836ad083" containerID="0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0" exitCode=0
Oct 30 22:12:37.106099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.105731    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4trf5"
Oct 30 22:12:37.106099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.105778    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4trf5" event={"ID":"ee472ef0-20a0-43e4-9b0a-760f836ad083","Type":"ContainerDied","Data":"0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0"}
Oct 30 22:12:37.106099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.105826    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4trf5" event={"ID":"ee472ef0-20a0-43e4-9b0a-760f836ad083","Type":"ContainerDied","Data":"57d0064d6279b967bbb2020cab1240b8afadc3c16e46817f212322a58fcad826"}
Oct 30 22:12:37.106099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.105853    4581 scope.go:122] "RemoveContainer" containerID="0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0"
Oct 30 22:12:37.115346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.115324    4581 scope.go:122] "RemoveContainer" containerID="0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0"
Oct 30 22:12:37.115621 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:12:37.115600    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0\": container with ID starting with 0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0 not found: ID does not exist" containerID="0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0"
Oct 30 22:12:37.115684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.115635    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0"} err="failed to get container status \"0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0\": rpc error: code = NotFound desc = could not find container \"0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0\": container with ID starting with 0e491a38f66f891d0f7fa314f276ed6ff85c76261bbb476abe920e917b382bf0 not found: ID does not exist"
Oct 30 22:12:37.149553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.149516    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4trf5"]
Oct 30 22:12:37.153865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.153835    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4trf5"]
Oct 30 22:12:37.164258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.164211    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bbd23474-9030-4f92-9e06-35ba36ab122e-manager-config\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.164412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.164268    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbd23474-9030-4f92-9e06-35ba36ab122e-cert\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.164412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.164351    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb84v\" (UniqueName: \"kubernetes.io/projected/bbd23474-9030-4f92-9e06-35ba36ab122e-kube-api-access-qb84v\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.164412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.164385    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd23474-9030-4f92-9e06-35ba36ab122e-metrics-certs\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.265652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.265612    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bbd23474-9030-4f92-9e06-35ba36ab122e-manager-config\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.265851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.265658    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbd23474-9030-4f92-9e06-35ba36ab122e-cert\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.265851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.265719    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb84v\" (UniqueName: \"kubernetes.io/projected/bbd23474-9030-4f92-9e06-35ba36ab122e-kube-api-access-qb84v\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.265851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.265761    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd23474-9030-4f92-9e06-35ba36ab122e-metrics-certs\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.266493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.266403    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bbd23474-9030-4f92-9e06-35ba36ab122e-manager-config\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.268401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.268377    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd23474-9030-4f92-9e06-35ba36ab122e-metrics-certs\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.268502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.268418    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbd23474-9030-4f92-9e06-35ba36ab122e-cert\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.283446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.283419    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb84v\" (UniqueName: \"kubernetes.io/projected/bbd23474-9030-4f92-9e06-35ba36ab122e-kube-api-access-qb84v\") pod \"jobset-controller-manager-6c87f6c66-mlkmg\" (UID: \"bbd23474-9030-4f92-9e06-35ba36ab122e\") " pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.350165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.350084    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:37.651047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:37.650959    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"]
Oct 30 22:12:37.655628 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:12:37.655594    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd23474_9030_4f92_9e06_35ba36ab122e.slice/crio-3823973f3294ddac994d497f497d387c2bcf7b2028a55641c0e819cf125cf7e5 WatchSource:0}: Error finding container 3823973f3294ddac994d497f497d387c2bcf7b2028a55641c0e819cf125cf7e5: Status 404 returned error can't find the container with id 3823973f3294ddac994d497f497d387c2bcf7b2028a55641c0e819cf125cf7e5
Oct 30 22:12:38.052254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:38.052162    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 22:12:38.094449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:38.094419    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee472ef0-20a0-43e4-9b0a-760f836ad083" path="/var/lib/kubelet/pods/ee472ef0-20a0-43e4-9b0a-760f836ad083/volumes"
Oct 30 22:12:38.113044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:38.113014    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg" event={"ID":"bbd23474-9030-4f92-9e06-35ba36ab122e","Type":"ContainerStarted","Data":"3823973f3294ddac994d497f497d387c2bcf7b2028a55641c0e819cf125cf7e5"}
Oct 30 22:12:40.815948 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:12:40.815909    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:12:41.130964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:41.130926    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg" event={"ID":"bbd23474-9030-4f92-9e06-35ba36ab122e","Type":"ContainerStarted","Data":"066c18c00d0fccf91a043619d8f9efcb95f85bd7a7ddd94c8949ce30650e34e2"}
Oct 30 22:12:41.131194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:41.131177    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:12:41.166289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:41.166202    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg" podStartSLOduration=1.772691054 podStartE2EDuration="4.166188227s" totalImagesPullingTime="2.393497173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:41.164805992 +0000 UTC m=+345.203744879" watchObservedRunningTime="2026-10-30 22:12:41.166188227 +0000 UTC m=+345.205127113"
Oct 30 22:12:52.141541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:12:52.141506    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg"
Oct 30 22:13:10.815706 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:13:10.815668    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:13:40.815563 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:13:40.815476    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:13:57.394948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.394909    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qq8k2"]
Oct 30 22:13:57.398880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.398853    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2"
Oct 30 22:13:57.410971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.410947    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xwk44\""
Oct 30 22:13:57.423736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.423668    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qq8k2"]
Oct 30 22:13:57.444903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.444869    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrkt\" (UniqueName: \"kubernetes.io/projected/cf0a521c-e2ab-4c4c-b20c-9d216d6959d9-kube-api-access-bbrkt\") pod \"authorino-6f6944bcdd-qq8k2\" (UID: \"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9\") " pod="kuadrant-system/authorino-6f6944bcdd-qq8k2"
Oct 30 22:13:57.546354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.546315    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrkt\" (UniqueName: \"kubernetes.io/projected/cf0a521c-e2ab-4c4c-b20c-9d216d6959d9-kube-api-access-bbrkt\") pod \"authorino-6f6944bcdd-qq8k2\" (UID: \"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9\") " pod="kuadrant-system/authorino-6f6944bcdd-qq8k2"
Oct 30 22:13:57.563773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.563735    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrkt\" (UniqueName: \"kubernetes.io/projected/cf0a521c-e2ab-4c4c-b20c-9d216d6959d9-kube-api-access-bbrkt\") pod \"authorino-6f6944bcdd-qq8k2\" (UID: \"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9\") " pod="kuadrant-system/authorino-6f6944bcdd-qq8k2"
Oct 30 22:13:57.666481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.666385    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qq8k2"]
Oct 30 22:13:57.666645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.666620    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2"
Oct 30 22:13:57.694826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.694786    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n2whf"]
Oct 30 22:13:57.698655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.698621    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:57.717853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.717822    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n2whf"]
Oct 30 22:13:57.748287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.747926    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qqk\" (UniqueName: \"kubernetes.io/projected/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-kube-api-access-92qqk\") pod \"authorino-6889cbd8c4-n2whf\" (UID: \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\") " pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:57.748287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.747996    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-tls-cert\") pod \"authorino-6889cbd8c4-n2whf\" (UID: \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\") " pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:57.757942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.757903    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 22:13:57.771041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.771009    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n2whf"]
Oct 30 22:13:57.771347 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:13:57.771262    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-92qqk tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-n2whf" podUID="87ea79cc-d85d-4de7-bdb0-420f76bbc4f0"
Oct 30 22:13:57.848902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.848863    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qqk\" (UniqueName: \"kubernetes.io/projected/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-kube-api-access-92qqk\") pod \"authorino-6889cbd8c4-n2whf\" (UID: \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\") " pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:57.849114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.848922    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-tls-cert\") pod \"authorino-6889cbd8c4-n2whf\" (UID: \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\") " pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:57.851794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.851761    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-tls-cert\") pod \"authorino-6889cbd8c4-n2whf\" (UID: \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\") " pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:57.865795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.865765    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qqk\" (UniqueName: \"kubernetes.io/projected/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-kube-api-access-92qqk\") pod \"authorino-6889cbd8c4-n2whf\" (UID: \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\") " pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:57.969156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:57.969058    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qq8k2"]
Oct 30 22:13:57.973302 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:13:57.973270    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf0a521c_e2ab_4c4c_b20c_9d216d6959d9.slice/crio-7e40897c4a11e277f7b925753b48e1012da68a6aa23696bb9db62c5773220d05 WatchSource:0}: Error finding container 7e40897c4a11e277f7b925753b48e1012da68a6aa23696bb9db62c5773220d05: Status 404 returned error can't find the container with id 7e40897c4a11e277f7b925753b48e1012da68a6aa23696bb9db62c5773220d05
Oct 30 22:13:58.535172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.535127    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2" event={"ID":"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9","Type":"ContainerStarted","Data":"7e40897c4a11e277f7b925753b48e1012da68a6aa23696bb9db62c5773220d05"}
Oct 30 22:13:58.535610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.535267    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:58.544180 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.544153    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:58.657052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.657014    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-kube-api-access-92qqk\" (UniqueName: \"kubernetes.io/projected/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-kube-api-access-92qqk\") pod \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\" (UID: \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\") "
Oct 30 22:13:58.657277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.657078    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-tls-cert\" (UniqueName: \"kubernetes.io/secret/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-tls-cert\") pod \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\" (UID: \"87ea79cc-d85d-4de7-bdb0-420f76bbc4f0\") "
Oct 30 22:13:58.659448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.659417    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-tls-cert" pod "87ea79cc-d85d-4de7-bdb0-420f76bbc4f0" (UID: "87ea79cc-d85d-4de7-bdb0-420f76bbc4f0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:58.659572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.659492    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-kube-api-access-92qqk" pod "87ea79cc-d85d-4de7-bdb0-420f76bbc4f0" (UID: "87ea79cc-d85d-4de7-bdb0-420f76bbc4f0"). InnerVolumeSpecName "kube-api-access-92qqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:58.758009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.757952    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92qqk\" (UniqueName: \"kubernetes.io/projected/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-kube-api-access-92qqk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:13:58.758009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:58.757996    4581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0-tls-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:13:59.545895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.545858    4581 generic.go:356] "Generic (PLEG): container finished" podID="1ccfd601-6d98-48b5-85e3-101586b02704" containerID="62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4" exitCode=0
Oct 30 22:13:59.546456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.545946    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" event={"ID":"1ccfd601-6d98-48b5-85e3-101586b02704","Type":"ContainerDied","Data":"62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4"}
Oct 30 22:13:59.546456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.546414    4581 scope.go:122] "RemoveContainer" containerID="62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4"
Oct 30 22:13:59.547670 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.547630    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:59.547774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.547716    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:59.547774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.547761    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:13:59.548032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.548010    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2" event={"ID":"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9","Type":"ContainerStarted","Data":"5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb"}
Oct 30 22:13:59.548087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.548019    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n2whf"
Oct 30 22:13:59.548142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.548093    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2" podUID="cf0a521c-e2ab-4c4c-b20c-9d216d6959d9" containerName="authorino" containerID="cri-o://5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb" gracePeriod=30
Oct 30 22:13:59.651234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.651184    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n2whf"]
Oct 30 22:13:59.656110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.656069    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n2whf"]
Oct 30 22:13:59.684896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.684840    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2" podStartSLOduration=1.897843899 podStartE2EDuration="2.684823918s" totalImagesPullingTime="786.980019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:59.684033431 +0000 UTC m=+423.722972316" watchObservedRunningTime="2026-10-30 22:13:59.684823918 +0000 UTC m=+423.723762803"
Oct 30 22:13:59.821440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.821408    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2"
Oct 30 22:13:59.869847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.869811    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0a521c-e2ab-4c4c-b20c-9d216d6959d9-kube-api-access-bbrkt\" (UniqueName: \"kubernetes.io/projected/cf0a521c-e2ab-4c4c-b20c-9d216d6959d9-kube-api-access-bbrkt\") pod \"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9\" (UID: \"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9\") "
Oct 30 22:13:59.872121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.872086    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0a521c-e2ab-4c4c-b20c-9d216d6959d9-kube-api-access-bbrkt" pod "cf0a521c-e2ab-4c4c-b20c-9d216d6959d9" (UID: "cf0a521c-e2ab-4c4c-b20c-9d216d6959d9"). InnerVolumeSpecName "kube-api-access-bbrkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:59.970785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:13:59.970747    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbrkt\" (UniqueName: \"kubernetes.io/projected/cf0a521c-e2ab-4c4c-b20c-9d216d6959d9-kube-api-access-bbrkt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:14:00.094498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.094417    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ea79cc-d85d-4de7-bdb0-420f76bbc4f0" path="/var/lib/kubelet/pods/87ea79cc-d85d-4de7-bdb0-420f76bbc4f0/volumes"
Oct 30 22:14:00.579203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.579150    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" event={"ID":"1ccfd601-6d98-48b5-85e3-101586b02704","Type":"ContainerStarted","Data":"93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8"}
Oct 30 22:14:00.580748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.580719    4581 generic.go:356] "Generic (PLEG): container finished" podID="cf0a521c-e2ab-4c4c-b20c-9d216d6959d9" containerID="5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb" exitCode=0
Oct 30 22:14:00.580890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.580754    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2"
Oct 30 22:14:00.580890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.580803    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2" event={"ID":"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9","Type":"ContainerDied","Data":"5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb"}
Oct 30 22:14:00.580890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.580840    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qq8k2" event={"ID":"cf0a521c-e2ab-4c4c-b20c-9d216d6959d9","Type":"ContainerDied","Data":"7e40897c4a11e277f7b925753b48e1012da68a6aa23696bb9db62c5773220d05"}
Oct 30 22:14:00.580890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.580859    4581 scope.go:122] "RemoveContainer" containerID="5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb"
Oct 30 22:14:00.590786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.590766    4581 scope.go:122] "RemoveContainer" containerID="5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb"
Oct 30 22:14:00.591103 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:14:00.591077    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb\": container with ID starting with 5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb not found: ID does not exist" containerID="5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb"
Oct 30 22:14:00.591233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.591129    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb"} err="failed to get container status \"5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb\": rpc error: code = NotFound desc = could not find container \"5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb\": container with ID starting with 5e1840fa4dabc7ab006406e2249f99f9b665269e6a321f17c4bf59b5caa032bb not found: ID does not exist"
Oct 30 22:14:00.678056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.677992    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qq8k2"]
Oct 30 22:14:00.679869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:00.679842    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qq8k2"]
Oct 30 22:14:01.377433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:01.377392    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:14:01.378787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:01.378756    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 22:14:01.378924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:01.378812    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 22:14:02.093516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:02.093484    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0a521c-e2ab-4c4c-b20c-9d216d6959d9" path="/var/lib/kubelet/pods/cf0a521c-e2ab-4c4c-b20c-9d216d6959d9/volumes"
Oct 30 22:14:02.377450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:02.377414    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 22:14:02.377635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:02.377471    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 22:14:03.272380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:03.272337    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:14:03.377830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:03.377789    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 22:14:03.378019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:03.377860    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 22:14:04.376764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:04.376726    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 22:14:04.377174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:04.376777    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 22:14:05.377783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:05.377738    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 22:14:05.378333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:05.377796    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 22:14:06.381297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:06.381257    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:14:06.382327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:14:06.382306    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:14:10.816372 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:14:10.816333    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:14:40.815048 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:14:40.814984    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:15:00.160763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.160707    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"]
Oct 30 22:15:00.161174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.161070    4581 cpu_manager.go:414] "containerMap: removing container" podUID="cf0a521c-e2ab-4c4c-b20c-9d216d6959d9" containerName="authorino"
Oct 30 22:15:00.161174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.161083    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0a521c-e2ab-4c4c-b20c-9d216d6959d9" containerName="authorino"
Oct 30 22:15:00.161290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.161180    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0a521c-e2ab-4c4c-b20c-9d216d6959d9" containerName="authorino"
Oct 30 22:15:00.165514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.165496    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:00.182368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.182344    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:15:00.187930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.187904    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:15:00.188087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.187940    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:15:00.190632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.190604    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"]
Oct 30 22:15:00.195357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.195320    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gn5\" (UniqueName: \"kubernetes.io/projected/0a36ae24-cd3e-43be-881d-ae177ab1c904-kube-api-access-j4gn5\") pod \"maas-api-key-cleanup-29889975-wnxwb\" (UID: \"0a36ae24-cd3e-43be-881d-ae177ab1c904\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:00.195501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.195371    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a36ae24-cd3e-43be-881d-ae177ab1c904-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-wnxwb\" (UID: \"0a36ae24-cd3e-43be-881d-ae177ab1c904\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:00.296505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.296472    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gn5\" (UniqueName: \"kubernetes.io/projected/0a36ae24-cd3e-43be-881d-ae177ab1c904-kube-api-access-j4gn5\") pod \"maas-api-key-cleanup-29889975-wnxwb\" (UID: \"0a36ae24-cd3e-43be-881d-ae177ab1c904\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:00.296696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.296523    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a36ae24-cd3e-43be-881d-ae177ab1c904-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-wnxwb\" (UID: \"0a36ae24-cd3e-43be-881d-ae177ab1c904\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:00.297207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.297184    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a36ae24-cd3e-43be-881d-ae177ab1c904-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-wnxwb\" (UID: \"0a36ae24-cd3e-43be-881d-ae177ab1c904\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:00.314057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.314029    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gn5\" (UniqueName: \"kubernetes.io/projected/0a36ae24-cd3e-43be-881d-ae177ab1c904-kube-api-access-j4gn5\") pod \"maas-api-key-cleanup-29889975-wnxwb\" (UID: \"0a36ae24-cd3e-43be-881d-ae177ab1c904\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:00.477948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.477865    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:00.775281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.775171    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"]
Oct 30 22:15:00.780633 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:15:00.780597    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a36ae24_cd3e_43be_881d_ae177ab1c904.slice/crio-69394a431f6b8581ef7401af1409df0ce430f49eb33c4f569bf6b5529f329c4d WatchSource:0}: Error finding container 69394a431f6b8581ef7401af1409df0ce430f49eb33c4f569bf6b5529f329c4d: Status 404 returned error can't find the container with id 69394a431f6b8581ef7401af1409df0ce430f49eb33c4f569bf6b5529f329c4d
Oct 30 22:15:00.889916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:00.889877    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" event={"ID":"0a36ae24-cd3e-43be-881d-ae177ab1c904","Type":"ContainerStarted","Data":"69394a431f6b8581ef7401af1409df0ce430f49eb33c4f569bf6b5529f329c4d"}
Oct 30 22:15:03.906999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:03.906962    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" event={"ID":"0a36ae24-cd3e-43be-881d-ae177ab1c904","Type":"ContainerStarted","Data":"35d494750180f85aac6f5bee1baae9da389cc66492e757fed10cfd69f4708853"}
Oct 30 22:15:03.941101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:03.941044    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" podStartSLOduration=1.414599261 podStartE2EDuration="3.941028618s" totalImagesPullingTime="2.526429357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.940108591 +0000 UTC m=+487.979047477" watchObservedRunningTime="2026-10-30 22:15:03.941028618 +0000 UTC m=+487.979967514"
Oct 30 22:15:04.913858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:04.913818    4581 generic.go:356] "Generic (PLEG): container finished" podID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerID="35d494750180f85aac6f5bee1baae9da389cc66492e757fed10cfd69f4708853" exitCode=60
Oct 30 22:15:04.914281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:04.913897    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" event={"ID":"0a36ae24-cd3e-43be-881d-ae177ab1c904","Type":"ContainerDied","Data":"35d494750180f85aac6f5bee1baae9da389cc66492e757fed10cfd69f4708853"}
Oct 30 22:15:04.914281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:04.914149    4581 scope.go:122] "RemoveContainer" containerID="35d494750180f85aac6f5bee1baae9da389cc66492e757fed10cfd69f4708853"
Oct 30 22:15:05.920765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:05.920669    4581 generic.go:356] "Generic (PLEG): container finished" podID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerID="66bb9a57668010c2be852467d6c98156689e33336b248d66bee6376b3cfac734" exitCode=60
Oct 30 22:15:05.920765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:05.920736    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" event={"ID":"0a36ae24-cd3e-43be-881d-ae177ab1c904","Type":"ContainerDied","Data":"66bb9a57668010c2be852467d6c98156689e33336b248d66bee6376b3cfac734"}
Oct 30 22:15:05.921359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:05.920794    4581 scope.go:122] "RemoveContainer" containerID="35d494750180f85aac6f5bee1baae9da389cc66492e757fed10cfd69f4708853"
Oct 30 22:15:05.921359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:05.921015    4581 scope.go:122] "RemoveContainer" containerID="66bb9a57668010c2be852467d6c98156689e33336b248d66bee6376b3cfac734"
Oct 30 22:15:05.921359 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:15:05.921259    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-wnxwb_odh-ai-gateway-infra(0a36ae24-cd3e-43be-881d-ae177ab1c904)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904"
Oct 30 22:15:06.927529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:06.927502    4581 scope.go:122] "RemoveContainer" containerID="66bb9a57668010c2be852467d6c98156689e33336b248d66bee6376b3cfac734"
Oct 30 22:15:06.927925 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:15:06.927717    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-wnxwb_odh-ai-gateway-infra(0a36ae24-cd3e-43be-881d-ae177ab1c904)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904"
Oct 30 22:15:10.816352 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:15:10.816312    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:15:20.088715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:20.088686    4581 scope.go:122] "RemoveContainer" containerID="66bb9a57668010c2be852467d6c98156689e33336b248d66bee6376b3cfac734"
Oct 30 22:15:21.004019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.003984    4581 generic.go:356] "Generic (PLEG): container finished" podID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerID="516ef46267ab1163f3a2e4d900e7dd60c713512ffbbd10ff2d4373e26a3a6329" exitCode=60
Oct 30 22:15:21.004182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.004071    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" event={"ID":"0a36ae24-cd3e-43be-881d-ae177ab1c904","Type":"ContainerDied","Data":"516ef46267ab1163f3a2e4d900e7dd60c713512ffbbd10ff2d4373e26a3a6329"}
Oct 30 22:15:21.004182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.004152    4581 scope.go:122] "RemoveContainer" containerID="66bb9a57668010c2be852467d6c98156689e33336b248d66bee6376b3cfac734"
Oct 30 22:15:21.004550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.004530    4581 scope.go:122] "RemoveContainer" containerID="516ef46267ab1163f3a2e4d900e7dd60c713512ffbbd10ff2d4373e26a3a6329"
Oct 30 22:15:21.004779 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:15:21.004752    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-wnxwb_odh-ai-gateway-infra(0a36ae24-cd3e-43be-881d-ae177ab1c904)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904"
Oct 30 22:15:21.079733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.079696    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"]
Oct 30 22:15:21.093266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.091411    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.109599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.109565    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"]
Oct 30 22:15:21.111857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.111834    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4qd26\""
Oct 30 22:15:21.112153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.112132    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 22:15:21.115449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.115425    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 22:15:21.115621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.115442    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 22:15:21.198155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.198122    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d4ed54-8bcf-442b-82c4-cbae47556237-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.198155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.198163    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.198469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.198189    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqv6b\" (UniqueName: \"kubernetes.io/projected/90d4ed54-8bcf-442b-82c4-cbae47556237-kube-api-access-jqv6b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.198469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.198325    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.198469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.198381    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.198469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.198413    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.299468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.299358    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d4ed54-8bcf-442b-82c4-cbae47556237-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.299468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.299428    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.299468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.299465    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqv6b\" (UniqueName: \"kubernetes.io/projected/90d4ed54-8bcf-442b-82c4-cbae47556237-kube-api-access-jqv6b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.299751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.299496    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.299751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.299524    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.299751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.299551    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.299944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.299916    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.300950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.300658    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.300950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.300692    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.302990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.302964    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d4ed54-8bcf-442b-82c4-cbae47556237-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.303425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.303380    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d4ed54-8bcf-442b-82c4-cbae47556237-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.316984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.316952    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqv6b\" (UniqueName: \"kubernetes.io/projected/90d4ed54-8bcf-442b-82c4-cbae47556237-kube-api-access-jqv6b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xdzkp\" (UID: \"90d4ed54-8bcf-442b-82c4-cbae47556237\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.419482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.419438    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:21.720164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:21.720131    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"]
Oct 30 22:15:21.723795 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:15:21.723768    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d4ed54_8bcf_442b_82c4_cbae47556237.slice/crio-89cc91c7218f84a8547c2e1dcdf3941eee29231821072dfab49ad46401683507 WatchSource:0}: Error finding container 89cc91c7218f84a8547c2e1dcdf3941eee29231821072dfab49ad46401683507: Status 404 returned error can't find the container with id 89cc91c7218f84a8547c2e1dcdf3941eee29231821072dfab49ad46401683507
Oct 30 22:15:22.012680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.012588    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp" event={"ID":"90d4ed54-8bcf-442b-82c4-cbae47556237","Type":"ContainerStarted","Data":"89cc91c7218f84a8547c2e1dcdf3941eee29231821072dfab49ad46401683507"}
Oct 30 22:15:22.049945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.049907    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"]
Oct 30 22:15:22.199593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.199566    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:22.310052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.309961    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a36ae24-cd3e-43be-881d-ae177ab1c904-kube-api-access-j4gn5\" (UniqueName: \"kubernetes.io/projected/0a36ae24-cd3e-43be-881d-ae177ab1c904-kube-api-access-j4gn5\") pod \"0a36ae24-cd3e-43be-881d-ae177ab1c904\" (UID: \"0a36ae24-cd3e-43be-881d-ae177ab1c904\") "
Oct 30 22:15:22.310052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.310021    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a36ae24-cd3e-43be-881d-ae177ab1c904-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a36ae24-cd3e-43be-881d-ae177ab1c904-maas-api-service-ca\") pod \"0a36ae24-cd3e-43be-881d-ae177ab1c904\" (UID: \"0a36ae24-cd3e-43be-881d-ae177ab1c904\") "
Oct 30 22:15:22.311023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.310989    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a36ae24-cd3e-43be-881d-ae177ab1c904-maas-api-service-ca" pod "0a36ae24-cd3e-43be-881d-ae177ab1c904" (UID: "0a36ae24-cd3e-43be-881d-ae177ab1c904"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:22.314319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.314257    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a36ae24-cd3e-43be-881d-ae177ab1c904-kube-api-access-j4gn5" pod "0a36ae24-cd3e-43be-881d-ae177ab1c904" (UID: "0a36ae24-cd3e-43be-881d-ae177ab1c904"). InnerVolumeSpecName "kube-api-access-j4gn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:22.411137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.411094    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gn5\" (UniqueName: \"kubernetes.io/projected/0a36ae24-cd3e-43be-881d-ae177ab1c904-kube-api-access-j4gn5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:15:22.411137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:22.411137    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a36ae24-cd3e-43be-881d-ae177ab1c904-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:15:23.023729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:23.023685    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"
Oct 30 22:15:23.023729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:23.023708    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb" event={"ID":"0a36ae24-cd3e-43be-881d-ae177ab1c904","Type":"ContainerDied","Data":"69394a431f6b8581ef7401af1409df0ce430f49eb33c4f569bf6b5529f329c4d"}
Oct 30 22:15:23.023985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:23.023781    4581 scope.go:122] "RemoveContainer" containerID="516ef46267ab1163f3a2e4d900e7dd60c713512ffbbd10ff2d4373e26a3a6329"
Oct 30 22:15:23.026282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:23.026247    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp" event={"ID":"90d4ed54-8bcf-442b-82c4-cbae47556237","Type":"ContainerStarted","Data":"917af2ca5457e9e0889e9184425e335b7911e8b67246a0f7f0e23c6534712e70"}
Oct 30 22:15:23.073887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:23.073836    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp" podStartSLOduration=0.926206698 podStartE2EDuration="2.073822357s" totalImagesPullingTime="1.147615659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:23.072250114 +0000 UTC m=+507.111189000" watchObservedRunningTime="2026-10-30 22:15:23.073822357 +0000 UTC m=+507.112761239"
Oct 30 22:15:23.102517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:23.102472    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"]
Oct 30 22:15:23.108260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:23.108228    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-wnxwb"]
Oct 30 22:15:23.420540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:23.420499    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:24.093900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:24.093865    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" path="/var/lib/kubelet/pods/0a36ae24-cd3e-43be-881d-ae177ab1c904/volumes"
Oct 30 22:15:26.419801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.419746    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"]
Oct 30 22:15:26.420324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.420305    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:26.420390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.420326    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:26.420390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.420340    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:26.420390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.420345    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:26.420390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.420358    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:26.420390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.420363    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:26.420560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.420430    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:26.420560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.420438    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:26.424853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.424833    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.446800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.446770    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 22:15:26.449235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.449189    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"]
Oct 30 22:15:26.545578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.545537    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.545578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.545579    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.545832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.545611    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgzc\" (UniqueName: \"kubernetes.io/projected/17847819-d7b1-4946-8a3c-326d50cdd08e-kube-api-access-2cgzc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.545832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.545728    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.545832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.545768    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.545832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.545790    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17847819-d7b1-4946-8a3c-326d50cdd08e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.646613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.646574    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.646613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.646618    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.646834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.646672    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgzc\" (UniqueName: \"kubernetes.io/projected/17847819-d7b1-4946-8a3c-326d50cdd08e-kube-api-access-2cgzc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.646834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.646704    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.646834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.646747    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.646834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.646771    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17847819-d7b1-4946-8a3c-326d50cdd08e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.647204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.647180    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.647330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.647193    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.647330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.647246    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.649167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.649145    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17847819-d7b1-4946-8a3c-326d50cdd08e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.649396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.649376    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17847819-d7b1-4946-8a3c-326d50cdd08e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.664119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.664085    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgzc\" (UniqueName: \"kubernetes.io/projected/17847819-d7b1-4946-8a3c-326d50cdd08e-kube-api-access-2cgzc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4\" (UID: \"17847819-d7b1-4946-8a3c-326d50cdd08e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:26.737041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:26.736946    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:27.040583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:27.040503    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"]
Oct 30 22:15:27.044551 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:15:27.044511    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17847819_d7b1_4946_8a3c_326d50cdd08e.slice/crio-546558e0e0fa40cb3361feb5f73b2606eaeead05e9dae147a43437b8a210d9e4 WatchSource:0}: Error finding container 546558e0e0fa40cb3361feb5f73b2606eaeead05e9dae147a43437b8a210d9e4: Status 404 returned error can't find the container with id 546558e0e0fa40cb3361feb5f73b2606eaeead05e9dae147a43437b8a210d9e4
Oct 30 22:15:28.056448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:28.056414    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4" event={"ID":"17847819-d7b1-4946-8a3c-326d50cdd08e","Type":"ContainerStarted","Data":"8e81405e296ddb578cc532968399adbbbc6a35d085d60a4ed745312b3de6bc68"}
Oct 30 22:15:28.056851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:28.056473    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4" event={"ID":"17847819-d7b1-4946-8a3c-326d50cdd08e","Type":"ContainerStarted","Data":"546558e0e0fa40cb3361feb5f73b2606eaeead05e9dae147a43437b8a210d9e4"}
Oct 30 22:15:28.091751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:28.091704    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4" podStartSLOduration=1.9128420240000001 podStartE2EDuration="2.091683471s" totalImagesPullingTime="178.841447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:28.091623283 +0000 UTC m=+512.130562171" watchObservedRunningTime="2026-10-30 22:15:28.091683471 +0000 UTC m=+512.130622357"
Oct 30 22:15:28.738070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:28.738021    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:31.420565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:31.420533    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:31.433363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:31.433331    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:32.100787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:32.100749    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp"
Oct 30 22:15:36.737159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:36.737121    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:36.749590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:36.749557    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:37.124086 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:37.124050    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4"
Oct 30 22:15:40.329665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.329626    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cd789d65d-jvcf8"]
Oct 30 22:15:40.330342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.330321    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a36ae24-cd3e-43be-881d-ae177ab1c904" containerName="cleanup"
Oct 30 22:15:40.333602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.333578    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd789d65d-jvcf8"
Oct 30 22:15:40.355012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.354980    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 22:15:40.355152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.355026    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xwk44\""
Oct 30 22:15:40.357438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.357414    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd789d65d-jvcf8"]
Oct 30 22:15:40.364260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.364203    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmz98\" (UniqueName: \"kubernetes.io/projected/cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec-kube-api-access-rmz98\") pod \"authorino-6cd789d65d-jvcf8\" (UID: \"cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec\") " pod="kuadrant-system/authorino-6cd789d65d-jvcf8"
Oct 30 22:15:40.364375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.364335    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec-tls-cert\") pod \"authorino-6cd789d65d-jvcf8\" (UID: \"cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec\") " pod="kuadrant-system/authorino-6cd789d65d-jvcf8"
Oct 30 22:15:40.465378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.465343    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmz98\" (UniqueName: \"kubernetes.io/projected/cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec-kube-api-access-rmz98\") pod \"authorino-6cd789d65d-jvcf8\" (UID: \"cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec\") " pod="kuadrant-system/authorino-6cd789d65d-jvcf8"
Oct 30 22:15:40.465550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.465400    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec-tls-cert\") pod \"authorino-6cd789d65d-jvcf8\" (UID: \"cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec\") " pod="kuadrant-system/authorino-6cd789d65d-jvcf8"
Oct 30 22:15:40.468107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.468072    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec-tls-cert\") pod \"authorino-6cd789d65d-jvcf8\" (UID: \"cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec\") " pod="kuadrant-system/authorino-6cd789d65d-jvcf8"
Oct 30 22:15:40.481844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.481817    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmz98\" (UniqueName: \"kubernetes.io/projected/cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec-kube-api-access-rmz98\") pod \"authorino-6cd789d65d-jvcf8\" (UID: \"cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec\") " pod="kuadrant-system/authorino-6cd789d65d-jvcf8"
Oct 30 22:15:40.646955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.646913    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd789d65d-jvcf8"
Oct 30 22:15:40.817616 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:15:40.817578    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:15:40.943161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:40.943075    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd789d65d-jvcf8"]
Oct 30 22:15:40.947156 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:15:40.947129    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf6b4c61_e0c0_48ea_9eec_d362e6fec6ec.slice/crio-44c44d24f13d3e65275191e8ca73a335e47522fc0c8582929fb3dac71ebb647f WatchSource:0}: Error finding container 44c44d24f13d3e65275191e8ca73a335e47522fc0c8582929fb3dac71ebb647f: Status 404 returned error can't find the container with id 44c44d24f13d3e65275191e8ca73a335e47522fc0c8582929fb3dac71ebb647f
Oct 30 22:15:41.134623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:41.134586    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd789d65d-jvcf8" event={"ID":"cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec","Type":"ContainerStarted","Data":"44c44d24f13d3e65275191e8ca73a335e47522fc0c8582929fb3dac71ebb647f"}
Oct 30 22:15:42.143620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:42.143572    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd789d65d-jvcf8" event={"ID":"cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec","Type":"ContainerStarted","Data":"996f4da9eff64851fbdc49e76a2a0acd2f6e11f9da7dd8dc4733a85d835739cb"}
Oct 30 22:15:42.181323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:42.181181    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6cd789d65d-jvcf8" podStartSLOduration=1.6347775279999999 podStartE2EDuration="2.181162605s" totalImagesPullingTime="546.385077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:42.179719908 +0000 UTC m=+526.218658797" watchObservedRunningTime="2026-10-30 22:15:42.181162605 +0000 UTC m=+526.220101493"
Oct 30 22:15:44.271649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.271611    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"]
Oct 30 22:15:44.275397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.275371    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:44.287482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.287451    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 22:15:44.296838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.296780    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-psbb2\""
Oct 30 22:15:44.300014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.299980    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"]
Oct 30 22:15:44.309360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.309331    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vsd\" (UniqueName: \"kubernetes.io/projected/d9267c00-aab9-4732-98c5-b5fc7eea8cf8-kube-api-access-68vsd\") pod \"payload-pre-processing-7b98bb678c-s6mwr\" (UID: \"d9267c00-aab9-4732-98c5-b5fc7eea8cf8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:44.309495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.309373    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9267c00-aab9-4732-98c5-b5fc7eea8cf8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s6mwr\" (UID: \"d9267c00-aab9-4732-98c5-b5fc7eea8cf8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:44.431944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.431900    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vsd\" (UniqueName: \"kubernetes.io/projected/d9267c00-aab9-4732-98c5-b5fc7eea8cf8-kube-api-access-68vsd\") pod \"payload-pre-processing-7b98bb678c-s6mwr\" (UID: \"d9267c00-aab9-4732-98c5-b5fc7eea8cf8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:44.432264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.432211    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9267c00-aab9-4732-98c5-b5fc7eea8cf8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s6mwr\" (UID: \"d9267c00-aab9-4732-98c5-b5fc7eea8cf8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:44.444242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.433327    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9267c00-aab9-4732-98c5-b5fc7eea8cf8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s6mwr\" (UID: \"d9267c00-aab9-4732-98c5-b5fc7eea8cf8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:44.449616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.449582    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vsd\" (UniqueName: \"kubernetes.io/projected/d9267c00-aab9-4732-98c5-b5fc7eea8cf8-kube-api-access-68vsd\") pod \"payload-pre-processing-7b98bb678c-s6mwr\" (UID: \"d9267c00-aab9-4732-98c5-b5fc7eea8cf8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:44.589501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.589392    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:44.892039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:44.892005    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"]
Oct 30 22:15:44.896476 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:15:44.896434    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9267c00_aab9_4732_98c5_b5fc7eea8cf8.slice/crio-328dfba8924ff83ce1cdcab8f506277ec202a602ffb8eb71aeef6cfe59457b32 WatchSource:0}: Error finding container 328dfba8924ff83ce1cdcab8f506277ec202a602ffb8eb71aeef6cfe59457b32: Status 404 returned error can't find the container with id 328dfba8924ff83ce1cdcab8f506277ec202a602ffb8eb71aeef6cfe59457b32
Oct 30 22:15:45.165506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:45.165392    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr" event={"ID":"d9267c00-aab9-4732-98c5-b5fc7eea8cf8","Type":"ContainerStarted","Data":"328dfba8924ff83ce1cdcab8f506277ec202a602ffb8eb71aeef6cfe59457b32"}
Oct 30 22:15:48.185802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:48.185760    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr" event={"ID":"d9267c00-aab9-4732-98c5-b5fc7eea8cf8","Type":"ContainerStarted","Data":"df0c606c890ead868ce55937fddbdac9fe9a718a111e7cf8d5a488c8eddbfcb2"}
Oct 30 22:15:48.186320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:48.185896    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:48.222614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:48.222549    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr" podStartSLOduration=1.71127747 podStartE2EDuration="4.22253603s" totalImagesPullingTime="2.51125856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:48.220329178 +0000 UTC m=+532.259268064" watchObservedRunningTime="2026-10-30 22:15:48.22253603 +0000 UTC m=+532.261474917"
Oct 30 22:15:50.880324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.880283    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"]
Oct 30 22:15:50.884178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.884148    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894014    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894055    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gcv\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-kube-api-access-n4gcv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894083    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e7f902-6226-48c4-a92e-b566be595f39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894301    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894351    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894504 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894479    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e7f902-6226-48c4-a92e-b566be595f39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894532    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894564    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.894618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.894590    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.910551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.910524    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"]
Oct 30 22:15:50.995031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.994987    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e7f902-6226-48c4-a92e-b566be595f39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.995031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995024    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.995378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995045    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.995378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995075    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.995378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995108    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.995378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995127    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gcv\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-kube-api-access-n4gcv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.995378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995151    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e7f902-6226-48c4-a92e-b566be595f39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.995378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995253    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.995378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995277    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.996020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995604    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.996020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995621    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.996020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.995886    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.996232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.996182    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e7f902-6226-48c4-a92e-b566be595f39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.996385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.996368    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.997744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.997724    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e7f902-6226-48c4-a92e-b566be595f39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:50.998730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:50.998713    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:51.013232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:51.013186    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:51.013424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:51.013328    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gcv\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-kube-api-access-n4gcv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sv4fp\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:51.201159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:51.201061    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:51.519986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:51.519890    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"]
Oct 30 22:15:51.523696 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:15:51.523670    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e7f902_6226_48c4_a92e_b566be595f39.slice/crio-91b2e35091ce519904f504a74030d97067560d89d8b6b8115baa10cf386fd07f WatchSource:0}: Error finding container 91b2e35091ce519904f504a74030d97067560d89d8b6b8115baa10cf386fd07f: Status 404 returned error can't find the container with id 91b2e35091ce519904f504a74030d97067560d89d8b6b8115baa10cf386fd07f
Oct 30 22:15:51.525908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:51.525873    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:51.525982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:51.525944    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:51.525982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:51.525977    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:15:52.210251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:52.210204    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" event={"ID":"75e7f902-6226-48c4-a92e-b566be595f39","Type":"ContainerStarted","Data":"1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a"}
Oct 30 22:15:52.210652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:52.210268    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" event={"ID":"75e7f902-6226-48c4-a92e-b566be595f39","Type":"ContainerStarted","Data":"91b2e35091ce519904f504a74030d97067560d89d8b6b8115baa10cf386fd07f"}
Oct 30 22:15:52.274706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:52.274633    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podStartSLOduration=2.274612537 podStartE2EDuration="2.274612537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:52.273579921 +0000 UTC m=+536.312518808" watchObservedRunningTime="2026-10-30 22:15:52.274612537 +0000 UTC m=+536.313551423"
Oct 30 22:15:53.201755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:53.201718    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:53.203254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:53.203205    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sv4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 22:15:53.203368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:53.203280    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 22:15:54.202031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.201999    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sv4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 22:15:54.202487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.202052    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 22:15:54.325325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.325289    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"]
Oct 30 22:15:54.331872 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.331844    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:54.348177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.348141    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"]
Oct 30 22:15:54.383567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.383526    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 22:15:54.383567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.383550    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 22:15:54.383567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.383554    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 22:15:54.383815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.383637    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 22:15:54.383815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.383695    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-vlvcv\""
Oct 30 22:15:54.425728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.425691    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4af176-ced2-4a38-93f8-cbe8130eaef2-metrics-certs\") pod \"ai-gateway-controller-7d5c88fbff-mvdht\" (UID: \"9e4af176-ced2-4a38-93f8-cbe8130eaef2\") " pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:54.425902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.425733    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqxqr\" (UniqueName: \"kubernetes.io/projected/9e4af176-ced2-4a38-93f8-cbe8130eaef2-kube-api-access-gqxqr\") pod \"ai-gateway-controller-7d5c88fbff-mvdht\" (UID: \"9e4af176-ced2-4a38-93f8-cbe8130eaef2\") " pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:54.526881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.526776    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4af176-ced2-4a38-93f8-cbe8130eaef2-metrics-certs\") pod \"ai-gateway-controller-7d5c88fbff-mvdht\" (UID: \"9e4af176-ced2-4a38-93f8-cbe8130eaef2\") " pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:54.526881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.526842    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqxqr\" (UniqueName: \"kubernetes.io/projected/9e4af176-ced2-4a38-93f8-cbe8130eaef2-kube-api-access-gqxqr\") pod \"ai-gateway-controller-7d5c88fbff-mvdht\" (UID: \"9e4af176-ced2-4a38-93f8-cbe8130eaef2\") " pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:54.529599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.529574    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4af176-ced2-4a38-93f8-cbe8130eaef2-metrics-certs\") pod \"ai-gateway-controller-7d5c88fbff-mvdht\" (UID: \"9e4af176-ced2-4a38-93f8-cbe8130eaef2\") " pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:54.543916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.543882    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqxqr\" (UniqueName: \"kubernetes.io/projected/9e4af176-ced2-4a38-93f8-cbe8130eaef2-kube-api-access-gqxqr\") pod \"ai-gateway-controller-7d5c88fbff-mvdht\" (UID: \"9e4af176-ced2-4a38-93f8-cbe8130eaef2\") " pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:54.648868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.648825    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:54.975658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:54.975604    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"]
Oct 30 22:15:54.980504 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:15:54.980474    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4af176_ced2_4a38_93f8_cbe8130eaef2.slice/crio-3411ff1a0432437f3cd5d49e7ab899fe5d0bf530db508f33dde12aaf33b92ef0 WatchSource:0}: Error finding container 3411ff1a0432437f3cd5d49e7ab899fe5d0bf530db508f33dde12aaf33b92ef0: Status 404 returned error can't find the container with id 3411ff1a0432437f3cd5d49e7ab899fe5d0bf530db508f33dde12aaf33b92ef0
Oct 30 22:15:55.201961 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:55.201919    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sv4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 22:15:55.202145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:55.201981    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 22:15:55.226782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:55.226689    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht" event={"ID":"9e4af176-ced2-4a38-93f8-cbe8130eaef2","Type":"ContainerStarted","Data":"3411ff1a0432437f3cd5d49e7ab899fe5d0bf530db508f33dde12aaf33b92ef0"}
Oct 30 22:15:56.202207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:56.202165    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sv4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 22:15:56.202688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:56.202246    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 22:15:57.202413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:57.202367    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sv4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 22:15:57.202905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:57.202430    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 22:15:57.719935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:57.719904    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 22:15:58.202638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:58.202601    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sv4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 22:15:58.203181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:58.202658    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 22:15:58.245799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:58.245755    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht" event={"ID":"9e4af176-ced2-4a38-93f8-cbe8130eaef2","Type":"ContainerStarted","Data":"ac42322d6882536ada139764c4231df9c156975512456ad9570f25753140c585"}
Oct 30 22:15:58.246003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:58.245902    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:15:58.281609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:58.281546    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht" podStartSLOduration=1.555886694 podStartE2EDuration="4.281532836s" totalImagesPullingTime="2.725646142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:58.279998367 +0000 UTC m=+542.318937252" watchObservedRunningTime="2026-10-30 22:15:58.281532836 +0000 UTC m=+542.320471721"
Oct 30 22:15:59.195161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:59.195131    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s6mwr"
Oct 30 22:15:59.804984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:59.804952    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:59.805479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:59.805456    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:15:59.806269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:15:59.806248    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:16:09.256392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:09.256360    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d5c88fbff-mvdht"
Oct 30 22:16:10.815724 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:16:10.815681    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:16:36.667270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.667007    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"]
Oct 30 22:16:36.678826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.678792    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.691378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.691350    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-09d0d5-openshift-default-dockercfg-x6h8g\""
Oct 30 22:16:36.704819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.704787    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"]
Oct 30 22:16:36.737829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.737797    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.738081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.737837    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.738081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.737856    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.738081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.737989    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btmjd\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-kube-api-access-btmjd\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.738081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.738039    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.738298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.738180    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.738298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.738247    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.738375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.738326    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.738375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.738363    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.839768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839724    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.839953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839798    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.839953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839819    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.839953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839846    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btmjd\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-kube-api-access-btmjd\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.839953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839876    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.839953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839913    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.839953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839937    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.840179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839968    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.840179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.839991    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.840283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.840169    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.840380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.840362    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.840586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.840570    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.841465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.841433    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.841630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.841558    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.843607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.843584    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.844770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.844721    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.861379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.861349    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.861614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.861591    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btmjd\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-kube-api-access-btmjd\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:36.935483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.935399    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"]
Oct 30 22:16:36.940479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.940453    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:36.955552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.955519    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-54114e-openshift-default-dockercfg-x4lxn\""
Oct 30 22:16:36.964425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.964398    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"]
Oct 30 22:16:36.994376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:36.994345    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041209    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041265    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxtg\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-kube-api-access-rzxtg\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041309    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041329    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041354    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041378    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041399    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041439    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.042254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.041468    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.142935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.142902    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.142935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.142943    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.142986    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143017    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143037    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143057    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxtg\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-kube-api-access-rzxtg\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143142    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143168    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143201    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143566    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143594    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143633    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143674    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.143965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.143946    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.145691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.145672    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.145921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.145906    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.166755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.166730    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxtg\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-kube-api-access-rzxtg\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.166992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.166975    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.210208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.210099    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"]
Oct 30 22:16:37.215722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.215691    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.244471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.244434    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.244718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.244502    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.244718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.244540    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.244879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.244737    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.244879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.244790    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.244879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.244833    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.245054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.244898    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ps2f\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-kube-api-access-8ps2f\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.245054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.244971    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.245054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.245022    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.246819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.246792    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"]
Oct 30 22:16:37.256113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.256089    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:37.269627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.269600    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-012ba0-openshift-default-dockercfg-gk2x2\""
Oct 30 22:16:37.320621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.320584    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"]
Oct 30 22:16:37.327133 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:37.327106    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9c3de2_5c80_4f68_9ab3_9885ce1a6175.slice/crio-cc06e5b55b79db67aaa6f8841cdc7d8e634fd48aceae2c2f96896f0a026063ec WatchSource:0}: Error finding container cc06e5b55b79db67aaa6f8841cdc7d8e634fd48aceae2c2f96896f0a026063ec: Status 404 returned error can't find the container with id cc06e5b55b79db67aaa6f8841cdc7d8e634fd48aceae2c2f96896f0a026063ec
Oct 30 22:16:37.329906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.329868    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.330010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.329954    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.330010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.329992    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.346749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.346720    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.346893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.346817    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.346893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.346861    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.346978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.346904    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.346978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.346933    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.347040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.347020    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.347073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.347049    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.347107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.347081    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.347153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.347136    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ps2f\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-kube-api-access-8ps2f\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.347485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.347460    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.347910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.347761    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.347910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.347777    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.348483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.348443    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.348789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.348765    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.351634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.351609    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.351724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.351666    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.371838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.371332    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ps2f\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-kube-api-access-8ps2f\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.373494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.373467    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.472870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.471661    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" event={"ID":"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175","Type":"ContainerStarted","Data":"2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712"}
Oct 30 22:16:37.472870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.471733    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" event={"ID":"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175","Type":"ContainerStarted","Data":"cc06e5b55b79db67aaa6f8841cdc7d8e634fd48aceae2c2f96896f0a026063ec"}
Oct 30 22:16:37.514386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.514323    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podStartSLOduration=1.514303769 podStartE2EDuration="1.514303769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:37.511898802 +0000 UTC m=+581.550837702" watchObservedRunningTime="2026-10-30 22:16:37.514303769 +0000 UTC m=+581.553242659"
Oct 30 22:16:37.532362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.532309    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"]
Oct 30 22:16:37.537107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.537067    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.542085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.541879    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:37.549182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549158    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.549315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549207    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.549315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549267    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c79349-2330-434d-8c58-39da2ad67c03-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.549315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549312    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c79349-2330-434d-8c58-39da2ad67c03-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.549423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549330    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.549423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549353    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbxb2\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-kube-api-access-xbxb2\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.549423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549372    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.549423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549389    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.549423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.549407    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.552238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.552181    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-421583-openshift-default-dockercfg-fp7vh\""
Oct 30 22:16:37.560905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.560876    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"]
Oct 30 22:16:37.574836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.574803    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"]
Oct 30 22:16:37.583840 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:37.583802    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc401bcec_d339_41e0_8a64_ec96d4fdc5eb.slice/crio-c50621bfce086e2510147f24c77e1a292938d54548d74112177e280c5dd66406 WatchSource:0}: Error finding container c50621bfce086e2510147f24c77e1a292938d54548d74112177e280c5dd66406: Status 404 returned error can't find the container with id c50621bfce086e2510147f24c77e1a292938d54548d74112177e280c5dd66406
Oct 30 22:16:37.586392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.586347    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.586495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.586449    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.586585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.586501    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.650956    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c79349-2330-434d-8c58-39da2ad67c03-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.651020    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c79349-2330-434d-8c58-39da2ad67c03-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.651116    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.651158    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbxb2\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-kube-api-access-xbxb2\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.651306    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.651344    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.651373    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.651491    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.651559    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.652028    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c79349-2330-434d-8c58-39da2ad67c03-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.652049    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.652557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.652325    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.653418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.653055    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.654179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.653763    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.679408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.679299    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.690178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.690114    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbxb2\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-kube-api-access-xbxb2\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.703896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.703829    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.712668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.712633    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c79349-2330-434d-8c58-39da2ad67c03-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.859948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.859910    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:37.883336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.883301    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"]
Oct 30 22:16:37.888723 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:37.888690    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaf4d3a_96f8_40c3_b0e8_03e6df1d6fc5.slice/crio-072b9d4d426a9c4a511a015fbb7b65012f332d349db6c2d88bd9e26dce6f776a WatchSource:0}: Error finding container 072b9d4d426a9c4a511a015fbb7b65012f332d349db6c2d88bd9e26dce6f776a: Status 404 returned error can't find the container with id 072b9d4d426a9c4a511a015fbb7b65012f332d349db6c2d88bd9e26dce6f776a
Oct 30 22:16:37.891869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.891829    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.891988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.891924    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.891988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.891970    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:37.994995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:37.994972    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:38.175562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.175481    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"]
Oct 30 22:16:38.178727 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:38.178688    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c79349_2330_434d_8c58_39da2ad67c03.slice/crio-b704f42538c84b542e857a2c89b496552357048674aece1895b4e3b673088ce1 WatchSource:0}: Error finding container b704f42538c84b542e857a2c89b496552357048674aece1895b4e3b673088ce1: Status 404 returned error can't find the container with id b704f42538c84b542e857a2c89b496552357048674aece1895b4e3b673088ce1
Oct 30 22:16:38.181336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.181298    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:38.181417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.181384    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:38.181456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.181426    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:16:38.480460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.480360    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" event={"ID":"16c79349-2330-434d-8c58-39da2ad67c03","Type":"ContainerStarted","Data":"f63623688584991420088798e71d800a8d2fab79b5b37d9724a008bbdb9a84be"}
Oct 30 22:16:38.480460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.480430    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" event={"ID":"16c79349-2330-434d-8c58-39da2ad67c03","Type":"ContainerStarted","Data":"b704f42538c84b542e857a2c89b496552357048674aece1895b4e3b673088ce1"}
Oct 30 22:16:38.482182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.482159    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" event={"ID":"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5","Type":"ContainerStarted","Data":"171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a"}
Oct 30 22:16:38.482337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.482206    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" event={"ID":"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5","Type":"ContainerStarted","Data":"072b9d4d426a9c4a511a015fbb7b65012f332d349db6c2d88bd9e26dce6f776a"}
Oct 30 22:16:38.484015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.483987    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" event={"ID":"c401bcec-d339-41e0-8a64-ec96d4fdc5eb","Type":"ContainerStarted","Data":"91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f"}
Oct 30 22:16:38.484117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.484042    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" event={"ID":"c401bcec-d339-41e0-8a64-ec96d4fdc5eb","Type":"ContainerStarted","Data":"c50621bfce086e2510147f24c77e1a292938d54548d74112177e280c5dd66406"}
Oct 30 22:16:38.519422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.519304    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" podStartSLOduration=1.5192855920000001 podStartE2EDuration="1.519285592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:38.517353487 +0000 UTC m=+582.556292373" watchObservedRunningTime="2026-10-30 22:16:38.519285592 +0000 UTC m=+582.558224479"
Oct 30 22:16:38.542575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.542546    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:38.547483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.547458    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:38.562938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.562895    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podStartSLOduration=2.562882009 podStartE2EDuration="2.562882009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:38.559528876 +0000 UTC m=+582.598467762" watchObservedRunningTime="2026-10-30 22:16:38.562882009 +0000 UTC m=+582.601820904"
Oct 30 22:16:38.599052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.598998    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podStartSLOduration=1.598984682 podStartE2EDuration="1.598984682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:38.59638165 +0000 UTC m=+582.635320537" watchObservedRunningTime="2026-10-30 22:16:38.598984682 +0000 UTC m=+582.637923568"
Oct 30 22:16:38.860554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.860521    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:38.999690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:38.999656    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:39.257346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.257263    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:39.263179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.263151    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:39.490262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.490199    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:39.490650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.490520    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:39.490650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.490557    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:39.490965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.490941    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:16:39.491104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.491015    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:16:39.491157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.491100    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:16:39.866157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:39.866126    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:40.495603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:40.495572    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:40.496524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:40.496503    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:16:40.816153 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:16:40.816072    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:16:44.534979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.534942    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"]
Oct 30 22:16:44.539052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.539030    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:44.551811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.551785    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-54114e\""
Oct 30 22:16:44.561528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.561505    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-54114e-dockercfg-b2fhr\""
Oct 30 22:16:44.565862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.565831    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"]
Oct 30 22:16:44.629259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.629204    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"]
Oct 30 22:16:44.633650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.633622    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:44.651442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.651411    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv\" (UID: \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:44.651832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.651808    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw9fl\" (UniqueName: \"kubernetes.io/projected/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-kube-api-access-jw9fl\") pod \"payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv\" (UID: \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:44.655902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.655880    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"]
Oct 30 22:16:44.752747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.752713    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngtn\" (UniqueName: \"kubernetes.io/projected/9be81c22-942d-4049-b971-e55f040bdd92-kube-api-access-xngtn\") pod \"payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc\" (UID: \"9be81c22-942d-4049-b971-e55f040bdd92\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:44.752931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.752802    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be81c22-942d-4049-b971-e55f040bdd92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc\" (UID: \"9be81c22-942d-4049-b971-e55f040bdd92\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:44.752931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.752852    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw9fl\" (UniqueName: \"kubernetes.io/projected/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-kube-api-access-jw9fl\") pod \"payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv\" (UID: \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:44.753011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.752983    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv\" (UID: \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:44.753737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.753717    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv\" (UID: \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:44.770248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.770180    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw9fl\" (UniqueName: \"kubernetes.io/projected/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-kube-api-access-jw9fl\") pod \"payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv\" (UID: \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:44.852917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.852824    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:44.854190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.854167    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngtn\" (UniqueName: \"kubernetes.io/projected/9be81c22-942d-4049-b971-e55f040bdd92-kube-api-access-xngtn\") pod \"payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc\" (UID: \"9be81c22-942d-4049-b971-e55f040bdd92\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:44.854293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.854200    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be81c22-942d-4049-b971-e55f040bdd92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc\" (UID: \"9be81c22-942d-4049-b971-e55f040bdd92\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:44.855022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.855002    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be81c22-942d-4049-b971-e55f040bdd92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc\" (UID: \"9be81c22-942d-4049-b971-e55f040bdd92\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:44.871434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.871410    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngtn\" (UniqueName: \"kubernetes.io/projected/9be81c22-942d-4049-b971-e55f040bdd92-kube-api-access-xngtn\") pod \"payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc\" (UID: \"9be81c22-942d-4049-b971-e55f040bdd92\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:44.958283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:44.958243    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:45.149418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.149336    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"]
Oct 30 22:16:45.152680 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:45.152651    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60eef6f3_d9eb_4716_bbb3_15f4e28249b8.slice/crio-4c89aa71500a65543c1b0b824a05b88e05b79c4455a129f9aa85130781ca6078 WatchSource:0}: Error finding container 4c89aa71500a65543c1b0b824a05b88e05b79c4455a129f9aa85130781ca6078: Status 404 returned error can't find the container with id 4c89aa71500a65543c1b0b824a05b88e05b79c4455a129f9aa85130781ca6078
Oct 30 22:16:45.250016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.249977    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"]
Oct 30 22:16:45.253775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.253748    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:45.264737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.264607    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"]
Oct 30 22:16:45.268188 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:45.268154    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be81c22_942d_4049_b971_e55f040bdd92.slice/crio-20002a512bf2591bf67791e353d601b1fdc00add37b5317ad886872dc951bb7f WatchSource:0}: Error finding container 20002a512bf2591bf67791e353d601b1fdc00add37b5317ad886872dc951bb7f: Status 404 returned error can't find the container with id 20002a512bf2591bf67791e353d601b1fdc00add37b5317ad886872dc951bb7f
Oct 30 22:16:45.268346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.268260    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d52c59\""
Oct 30 22:16:45.273896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.273874    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"]
Oct 30 22:16:45.358428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.358393    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hzz\" (UniqueName: \"kubernetes.io/projected/14657837-22d6-40d8-af6f-a6126c9ae497-kube-api-access-q5hzz\") pod \"payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8\" (UID: \"14657837-22d6-40d8-af6f-a6126c9ae497\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:45.358428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.358434    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14657837-22d6-40d8-af6f-a6126c9ae497-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8\" (UID: \"14657837-22d6-40d8-af6f-a6126c9ae497\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:45.459304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.459153    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hzz\" (UniqueName: \"kubernetes.io/projected/14657837-22d6-40d8-af6f-a6126c9ae497-kube-api-access-q5hzz\") pod \"payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8\" (UID: \"14657837-22d6-40d8-af6f-a6126c9ae497\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:45.459304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.459205    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14657837-22d6-40d8-af6f-a6126c9ae497-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8\" (UID: \"14657837-22d6-40d8-af6f-a6126c9ae497\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:45.459988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.459956    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14657837-22d6-40d8-af6f-a6126c9ae497-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8\" (UID: \"14657837-22d6-40d8-af6f-a6126c9ae497\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:45.479087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.479051    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hzz\" (UniqueName: \"kubernetes.io/projected/14657837-22d6-40d8-af6f-a6126c9ae497-kube-api-access-q5hzz\") pod \"payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8\" (UID: \"14657837-22d6-40d8-af6f-a6126c9ae497\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:45.527313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.527276    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc" event={"ID":"9be81c22-942d-4049-b971-e55f040bdd92","Type":"ContainerStarted","Data":"9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09"}
Oct 30 22:16:45.527512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.527339    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc" event={"ID":"9be81c22-942d-4049-b971-e55f040bdd92","Type":"ContainerStarted","Data":"20002a512bf2591bf67791e353d601b1fdc00add37b5317ad886872dc951bb7f"}
Oct 30 22:16:45.527512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.527492    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:45.529070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.529048    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv" event={"ID":"60eef6f3-d9eb-4716-bbb3-15f4e28249b8","Type":"ContainerStarted","Data":"0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0"}
Oct 30 22:16:45.529196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.529092    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv" event={"ID":"60eef6f3-d9eb-4716-bbb3-15f4e28249b8","Type":"ContainerStarted","Data":"4c89aa71500a65543c1b0b824a05b88e05b79c4455a129f9aa85130781ca6078"}
Oct 30 22:16:45.529196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.529134    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:45.568762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.568727    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:45.594249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.594180    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc" podStartSLOduration=1.594157783 podStartE2EDuration="1.594157783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:45.560510099 +0000 UTC m=+589.599448985" watchObservedRunningTime="2026-10-30 22:16:45.594157783 +0000 UTC m=+589.633096671"
Oct 30 22:16:45.595585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.595530    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv" podStartSLOduration=1.595515427 podStartE2EDuration="1.595515427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:45.592643122 +0000 UTC m=+589.631582009" watchObservedRunningTime="2026-10-30 22:16:45.595515427 +0000 UTC m=+589.634454319"
Oct 30 22:16:45.866208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:45.866174    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"]
Oct 30 22:16:45.869185 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:45.869152    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14657837_22d6_40d8_af6f_a6126c9ae497.slice/crio-cb26459bc9157d5665d4701dd068da0d2e19e877d548db4bd390c503f8313518 WatchSource:0}: Error finding container cb26459bc9157d5665d4701dd068da0d2e19e877d548db4bd390c503f8313518: Status 404 returned error can't find the container with id cb26459bc9157d5665d4701dd068da0d2e19e877d548db4bd390c503f8313518
Oct 30 22:16:46.541328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.541288    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8" event={"ID":"14657837-22d6-40d8-af6f-a6126c9ae497","Type":"ContainerStarted","Data":"07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433"}
Oct 30 22:16:46.541539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.541383    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8" event={"ID":"14657837-22d6-40d8-af6f-a6126c9ae497","Type":"ContainerStarted","Data":"cb26459bc9157d5665d4701dd068da0d2e19e877d548db4bd390c503f8313518"}
Oct 30 22:16:46.541539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.541481    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:46.581556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.581478    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8" podStartSLOduration=1.5814582659999998 podStartE2EDuration="1.581458266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:46.579723191 +0000 UTC m=+590.618662078" watchObservedRunningTime="2026-10-30 22:16:46.581458266 +0000 UTC m=+590.620397149"
Oct 30 22:16:46.709893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.709855    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"]
Oct 30 22:16:46.713656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.713633    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:46.726132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.726099    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0ad7c5\""
Oct 30 22:16:46.727790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.727756    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"]
Oct 30 22:16:46.731460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.731440    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:46.739459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.737671    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"]
Oct 30 22:16:46.743533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.743505    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0ad7c5-dockercfg-hwjhj\""
Oct 30 22:16:46.765733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.765700    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"]
Oct 30 22:16:46.772802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.772766    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879ed01f-aaea-42f9-86dd-eb467e927802-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz\" (UID: \"879ed01f-aaea-42f9-86dd-eb467e927802\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:46.772992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.772893    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29m5l\" (UniqueName: \"kubernetes.io/projected/879ed01f-aaea-42f9-86dd-eb467e927802-kube-api-access-29m5l\") pod \"payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz\" (UID: \"879ed01f-aaea-42f9-86dd-eb467e927802\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:46.772992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.772919    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjjh\" (UniqueName: \"kubernetes.io/projected/9f048de9-6d24-4561-bfcd-d0501b41bbf2-kube-api-access-rtjjh\") pod \"payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp\" (UID: \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:46.773128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.773006    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f048de9-6d24-4561-bfcd-d0501b41bbf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp\" (UID: \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:46.874630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.874592    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879ed01f-aaea-42f9-86dd-eb467e927802-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz\" (UID: \"879ed01f-aaea-42f9-86dd-eb467e927802\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:46.874836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.874687    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29m5l\" (UniqueName: \"kubernetes.io/projected/879ed01f-aaea-42f9-86dd-eb467e927802-kube-api-access-29m5l\") pod \"payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz\" (UID: \"879ed01f-aaea-42f9-86dd-eb467e927802\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:46.874836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.874711    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjjh\" (UniqueName: \"kubernetes.io/projected/9f048de9-6d24-4561-bfcd-d0501b41bbf2-kube-api-access-rtjjh\") pod \"payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp\" (UID: \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:46.874836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.874727    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f048de9-6d24-4561-bfcd-d0501b41bbf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp\" (UID: \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:46.875518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.875493    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879ed01f-aaea-42f9-86dd-eb467e927802-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz\" (UID: \"879ed01f-aaea-42f9-86dd-eb467e927802\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:46.875617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.875497    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f048de9-6d24-4561-bfcd-d0501b41bbf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp\" (UID: \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:46.915255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.912448    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29m5l\" (UniqueName: \"kubernetes.io/projected/879ed01f-aaea-42f9-86dd-eb467e927802-kube-api-access-29m5l\") pod \"payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz\" (UID: \"879ed01f-aaea-42f9-86dd-eb467e927802\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:46.924000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:46.923965    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjjh\" (UniqueName: \"kubernetes.io/projected/9f048de9-6d24-4561-bfcd-d0501b41bbf2-kube-api-access-rtjjh\") pod \"payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp\" (UID: \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:47.029891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.029858    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:47.048613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.048572    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:47.334299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.334265    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"]
Oct 30 22:16:47.337762 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:47.337735    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f048de9_6d24_4561_bfcd_d0501b41bbf2.slice/crio-e6d2858de1a4b231356686a7d9d7c078a3e9c29f49b266bac7e1a93c708dc796 WatchSource:0}: Error finding container e6d2858de1a4b231356686a7d9d7c078a3e9c29f49b266bac7e1a93c708dc796: Status 404 returned error can't find the container with id e6d2858de1a4b231356686a7d9d7c078a3e9c29f49b266bac7e1a93c708dc796
Oct 30 22:16:47.371401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.371368    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"]
Oct 30 22:16:47.377325 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:47.375956    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879ed01f_aaea_42f9_86dd_eb467e927802.slice/crio-17ff64dfac8c3468296ac44bad157377e96c7cfc8e7fa26c9ce343bd70a727e0 WatchSource:0}: Error finding container 17ff64dfac8c3468296ac44bad157377e96c7cfc8e7fa26c9ce343bd70a727e0: Status 404 returned error can't find the container with id 17ff64dfac8c3468296ac44bad157377e96c7cfc8e7fa26c9ce343bd70a727e0
Oct 30 22:16:47.550059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.550023    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz" event={"ID":"879ed01f-aaea-42f9-86dd-eb467e927802","Type":"ContainerStarted","Data":"80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff"}
Oct 30 22:16:47.550280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.550093    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:16:47.550280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.550117    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz" event={"ID":"879ed01f-aaea-42f9-86dd-eb467e927802","Type":"ContainerStarted","Data":"17ff64dfac8c3468296ac44bad157377e96c7cfc8e7fa26c9ce343bd70a727e0"}
Oct 30 22:16:47.551941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.551907    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp" event={"ID":"9f048de9-6d24-4561-bfcd-d0501b41bbf2","Type":"ContainerStarted","Data":"b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407"}
Oct 30 22:16:47.552100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.551971    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:47.552100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.551995    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp" event={"ID":"9f048de9-6d24-4561-bfcd-d0501b41bbf2","Type":"ContainerStarted","Data":"e6d2858de1a4b231356686a7d9d7c078a3e9c29f49b266bac7e1a93c708dc796"}
Oct 30 22:16:47.583857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.583802    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz" podStartSLOduration=1.5837868080000002 podStartE2EDuration="1.583786808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:47.582835816 +0000 UTC m=+591.621774703" watchObservedRunningTime="2026-10-30 22:16:47.583786808 +0000 UTC m=+591.622725732"
Oct 30 22:16:47.619749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:47.619683    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp" podStartSLOduration=1.6196631099999999 podStartE2EDuration="1.61966311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:47.617291095 +0000 UTC m=+591.656229982" watchObservedRunningTime="2026-10-30 22:16:47.61966311 +0000 UTC m=+591.658601997"
Oct 30 22:16:49.689702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.689663    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"]
Oct 30 22:16:49.693624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.693605    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:49.714934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.714903    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"]
Oct 30 22:16:49.719662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.719626    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-421583\""
Oct 30 22:16:49.719998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.719959    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-421583-dockercfg-swp2j\""
Oct 30 22:16:49.792663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.792632    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"]
Oct 30 22:16:49.796374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.796354    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:49.809716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.809678    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6g8\" (UniqueName: \"kubernetes.io/projected/7c726e47-afbb-478a-a30e-91f5177e570c-kube-api-access-ws6g8\") pod \"payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj\" (UID: \"7c726e47-afbb-478a-a30e-91f5177e570c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:49.809879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.809830    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c726e47-afbb-478a-a30e-91f5177e570c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj\" (UID: \"7c726e47-afbb-478a-a30e-91f5177e570c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:49.837368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.837330    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"]
Oct 30 22:16:49.911978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.911929    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6g8\" (UniqueName: \"kubernetes.io/projected/7c726e47-afbb-478a-a30e-91f5177e570c-kube-api-access-ws6g8\") pod \"payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj\" (UID: \"7c726e47-afbb-478a-a30e-91f5177e570c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:49.912366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.912342    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709c3ea4-f775-4be6-8d58-4bcaa7b53277-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m\" (UID: \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:49.912598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.912580    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c726e47-afbb-478a-a30e-91f5177e570c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj\" (UID: \"7c726e47-afbb-478a-a30e-91f5177e570c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:49.912801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.912786    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g27f\" (UniqueName: \"kubernetes.io/projected/709c3ea4-f775-4be6-8d58-4bcaa7b53277-kube-api-access-7g27f\") pod \"payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m\" (UID: \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:49.917146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.917115    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c726e47-afbb-478a-a30e-91f5177e570c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj\" (UID: \"7c726e47-afbb-478a-a30e-91f5177e570c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:49.986974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:49.986886    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6g8\" (UniqueName: \"kubernetes.io/projected/7c726e47-afbb-478a-a30e-91f5177e570c-kube-api-access-ws6g8\") pod \"payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj\" (UID: \"7c726e47-afbb-478a-a30e-91f5177e570c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:50.009181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.009145    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:50.014025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.013995    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709c3ea4-f775-4be6-8d58-4bcaa7b53277-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m\" (UID: \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:50.014182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.014140    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g27f\" (UniqueName: \"kubernetes.io/projected/709c3ea4-f775-4be6-8d58-4bcaa7b53277-kube-api-access-7g27f\") pod \"payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m\" (UID: \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:50.015141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.015117    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709c3ea4-f775-4be6-8d58-4bcaa7b53277-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m\" (UID: \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:50.032137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.032049    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g27f\" (UniqueName: \"kubernetes.io/projected/709c3ea4-f775-4be6-8d58-4bcaa7b53277-kube-api-access-7g27f\") pod \"payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m\" (UID: \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:50.115769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.115376    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:50.321409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.321327    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"]
Oct 30 22:16:50.324534 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:50.324510    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c726e47_afbb_478a_a30e_91f5177e570c.slice/crio-aa86952ed7daac79033fbe99bf919401802410dcbb7b96b19e5d56aa23f1d611 WatchSource:0}: Error finding container aa86952ed7daac79033fbe99bf919401802410dcbb7b96b19e5d56aa23f1d611: Status 404 returned error can't find the container with id aa86952ed7daac79033fbe99bf919401802410dcbb7b96b19e5d56aa23f1d611
Oct 30 22:16:50.435987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.435953    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"]
Oct 30 22:16:50.439812 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:50.439782    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709c3ea4_f775_4be6_8d58_4bcaa7b53277.slice/crio-5e9bc00fc2a1a2818457b3b7adba06833bff45c9fe0742b5b7c772c8e57a2ba8 WatchSource:0}: Error finding container 5e9bc00fc2a1a2818457b3b7adba06833bff45c9fe0742b5b7c772c8e57a2ba8: Status 404 returned error can't find the container with id 5e9bc00fc2a1a2818457b3b7adba06833bff45c9fe0742b5b7c772c8e57a2ba8
Oct 30 22:16:50.508611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.508579    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"]
Oct 30 22:16:50.513871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.513849    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:50.531962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.531916    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-2b1e88\""
Oct 30 22:16:50.532634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.532604    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"]
Oct 30 22:16:50.569722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.569680    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"]
Oct 30 22:16:50.573659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.573593    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:50.587073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.585493    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj" event={"ID":"7c726e47-afbb-478a-a30e-91f5177e570c","Type":"ContainerStarted","Data":"23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215"}
Oct 30 22:16:50.587073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.585574    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj" event={"ID":"7c726e47-afbb-478a-a30e-91f5177e570c","Type":"ContainerStarted","Data":"aa86952ed7daac79033fbe99bf919401802410dcbb7b96b19e5d56aa23f1d611"}
Oct 30 22:16:50.587073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.585721    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:16:50.588544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.588357    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-2b1e88-dockercfg-wh6bw\""
Oct 30 22:16:50.589578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.589548    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m" event={"ID":"709c3ea4-f775-4be6-8d58-4bcaa7b53277","Type":"ContainerStarted","Data":"43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40"}
Oct 30 22:16:50.589720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.589610    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m" event={"ID":"709c3ea4-f775-4be6-8d58-4bcaa7b53277","Type":"ContainerStarted","Data":"5e9bc00fc2a1a2818457b3b7adba06833bff45c9fe0742b5b7c772c8e57a2ba8"}
Oct 30 22:16:50.589720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.589658    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:16:50.602652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.602623    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"]
Oct 30 22:16:50.619828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.619798    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd\" (UID: \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:50.619975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.619859    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4zd\" (UniqueName: \"kubernetes.io/projected/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-kube-api-access-hj4zd\") pod \"payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd\" (UID: \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:50.619975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.619909    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgsw\" (UniqueName: \"kubernetes.io/projected/9adb45cf-8bab-4ea3-bb70-9af14c11453d-kube-api-access-crgsw\") pod \"payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v\" (UID: \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:50.620060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.619994    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adb45cf-8bab-4ea3-bb70-9af14c11453d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v\" (UID: \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:50.691209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.691147    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m" podStartSLOduration=1.691123203 podStartE2EDuration="1.691123203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:50.655558761 +0000 UTC m=+594.694497648" watchObservedRunningTime="2026-10-30 22:16:50.691123203 +0000 UTC m=+594.730062090"
Oct 30 22:16:50.721195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.721155    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adb45cf-8bab-4ea3-bb70-9af14c11453d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v\" (UID: \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:50.721427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.721384    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd\" (UID: \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:50.721502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.721476    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4zd\" (UniqueName: \"kubernetes.io/projected/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-kube-api-access-hj4zd\") pod \"payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd\" (UID: \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:50.721561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.721505    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgsw\" (UniqueName: \"kubernetes.io/projected/9adb45cf-8bab-4ea3-bb70-9af14c11453d-kube-api-access-crgsw\") pod \"payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v\" (UID: \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:50.722388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.722361    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd\" (UID: \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:50.722485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.722446    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adb45cf-8bab-4ea3-bb70-9af14c11453d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v\" (UID: \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:50.742618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.742579    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4zd\" (UniqueName: \"kubernetes.io/projected/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-kube-api-access-hj4zd\") pod \"payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd\" (UID: \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:50.742779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.742579    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgsw\" (UniqueName: \"kubernetes.io/projected/9adb45cf-8bab-4ea3-bb70-9af14c11453d-kube-api-access-crgsw\") pod \"payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v\" (UID: \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:50.827665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.827576    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:50.893634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:50.893318    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:51.181592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.181517    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj" podStartSLOduration=2.181489968 podStartE2EDuration="2.181489968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:50.692527129 +0000 UTC m=+594.731466016" watchObservedRunningTime="2026-10-30 22:16:51.181489968 +0000 UTC m=+595.220428855"
Oct 30 22:16:51.189950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.183580    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"]
Oct 30 22:16:51.189950 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:51.187113    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0b8bab_0940_4b75_b95b_f5a167ac19bf.slice/crio-93a4056511d1b7b8d6857fd126bc829e3951e68bc633ae1f5540f8f02b14f23e WatchSource:0}: Error finding container 93a4056511d1b7b8d6857fd126bc829e3951e68bc633ae1f5540f8f02b14f23e: Status 404 returned error can't find the container with id 93a4056511d1b7b8d6857fd126bc829e3951e68bc633ae1f5540f8f02b14f23e
Oct 30 22:16:51.234768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.234740    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"]
Oct 30 22:16:51.238442 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:16:51.238409    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adb45cf_8bab_4ea3_bb70_9af14c11453d.slice/crio-8d884a05d835ee1a936fa7de6262bd0f974aa82036ca0a1a8b6a275549ac49a7 WatchSource:0}: Error finding container 8d884a05d835ee1a936fa7de6262bd0f974aa82036ca0a1a8b6a275549ac49a7: Status 404 returned error can't find the container with id 8d884a05d835ee1a936fa7de6262bd0f974aa82036ca0a1a8b6a275549ac49a7
Oct 30 22:16:51.601076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.600979    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd" event={"ID":"6d0b8bab-0940-4b75-b95b-f5a167ac19bf","Type":"ContainerStarted","Data":"2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777"}
Oct 30 22:16:51.601355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.601335    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd" event={"ID":"6d0b8bab-0940-4b75-b95b-f5a167ac19bf","Type":"ContainerStarted","Data":"93a4056511d1b7b8d6857fd126bc829e3951e68bc633ae1f5540f8f02b14f23e"}
Oct 30 22:16:51.601727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.601653    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:16:51.603628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.603601    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v" event={"ID":"9adb45cf-8bab-4ea3-bb70-9af14c11453d","Type":"ContainerStarted","Data":"8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af"}
Oct 30 22:16:51.603830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.603654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v" event={"ID":"9adb45cf-8bab-4ea3-bb70-9af14c11453d","Type":"ContainerStarted","Data":"8d884a05d835ee1a936fa7de6262bd0f974aa82036ca0a1a8b6a275549ac49a7"}
Oct 30 22:16:51.645512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.645440    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd" podStartSLOduration=1.6454186800000001 podStartE2EDuration="1.64541868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:51.642342165 +0000 UTC m=+595.681281052" watchObservedRunningTime="2026-10-30 22:16:51.64541868 +0000 UTC m=+595.684357570"
Oct 30 22:16:51.681247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:51.680820    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v" podStartSLOduration=1.680798315 podStartE2EDuration="1.680798315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:51.679627681 +0000 UTC m=+595.718566567" watchObservedRunningTime="2026-10-30 22:16:51.680798315 +0000 UTC m=+595.719737202"
Oct 30 22:16:52.609610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:52.609577    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:16:56.325692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:56.325599    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:16:56.328852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:56.328820    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:16:56.545356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:56.545330    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 22:16:56.545480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:56.545409    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 22:16:57.578508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:57.578478    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 22:16:58.576409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:58.576372    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:16:58.579587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:16:58.579558    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:17:01.607514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:01.607485    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 22:17:01.607915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:01.607575    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 22:17:01.713153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:01.713118    4581 generic.go:356] "Generic (PLEG): container finished" podID="16c79349-2330-434d-8c58-39da2ad67c03" containerID="f63623688584991420088798e71d800a8d2fab79b5b37d9724a008bbdb9a84be" exitCode=0
Oct 30 22:17:01.713373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:01.713202    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" event={"ID":"16c79349-2330-434d-8c58-39da2ad67c03","Type":"ContainerDied","Data":"f63623688584991420088798e71d800a8d2fab79b5b37d9724a008bbdb9a84be"}
Oct 30 22:17:01.713724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:01.713708    4581 scope.go:122] "RemoveContainer" containerID="f63623688584991420088798e71d800a8d2fab79b5b37d9724a008bbdb9a84be"
Oct 30 22:17:01.715188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:01.715158    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:01.715323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:01.715272    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:01.715389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:01.715327    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:02.620763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:02.620736    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 22:17:02.774245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:02.754909    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" event={"ID":"16c79349-2330-434d-8c58-39da2ad67c03","Type":"ContainerStarted","Data":"a0ff16d26caf5351a73c1606c1ed702291ea5ff90a719b638efdd00092954663"}
Oct 30 22:17:02.868245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:02.861658    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:17:02.868245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:02.861792    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:17:02.868245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:02.861837    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 22:17:03.620863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:03.620256    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 22:17:03.914713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:03.914617    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:17:03.914987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:03.914954    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 22:17:04.866244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:04.864621    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:17:04.866244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:04.864683    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 22:17:05.863063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:05.862628    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:17:05.863063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:05.862690    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 22:17:06.863250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:06.861716    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:17:06.863250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:06.861795    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 22:17:07.864242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:07.860998    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:17:07.864242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:07.861073    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 22:17:08.225765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.225669    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"]
Oct 30 22:17:08.235518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.235488    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.244034 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.243998    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"]
Oct 30 22:17:08.248975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.248465    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-dshm\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.248975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.248532    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-home\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.248975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.248669    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpkc\" (UniqueName: \"kubernetes.io/projected/46bf79be-6e7c-4780-9d50-5d13d5d57bae-kube-api-access-6cpkc\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.248975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.248719    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-model-cache\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.248975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.248751    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tmp-dir\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.248975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.248788    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tls-certs\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.319657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.291621    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"simulator-w2-421583-kserve-self-signed-certs\""
Oct 30 22:17:08.319657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.291708    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"openshift-service-ca.crt\""
Oct 30 22:17:08.319657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.291826    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"default-dockercfg-vhmms\""
Oct 30 22:17:08.319657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.291935    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"kube-root-ca.crt\""
Oct 30 22:17:08.349939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.349298    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-model-cache\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.349939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.349337    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tmp-dir\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.349939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.349357    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tls-certs\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.349939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.349395    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-dshm\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.349939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.349439    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-home\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.349939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.349567    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpkc\" (UniqueName: \"kubernetes.io/projected/46bf79be-6e7c-4780-9d50-5d13d5d57bae-kube-api-access-6cpkc\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.349939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.349769    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tmp-dir\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.349939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.349876    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-model-cache\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.350587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.350020    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-home\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.422437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.422370    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-dshm\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.425041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.424977    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpkc\" (UniqueName: \"kubernetes.io/projected/46bf79be-6e7c-4780-9d50-5d13d5d57bae-kube-api-access-6cpkc\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.426189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.426162    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tls-certs\") pod \"simulator-w2-421583-kserve-6c8f496b58-s75q7\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") " pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.629694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.629453    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:08.886739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.886663    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:17:08.887374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.887355    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:17:08.888426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:08.888398    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:17:09.025698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:09.025655    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"]
Oct 30 22:17:09.027039 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:17:09.027009    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46bf79be_6e7c_4780_9d50_5d13d5d57bae.slice/crio-08637c82cddc68dd2aeea753c7f101927c47080bebf90f8b3299d24c6d52aa31 WatchSource:0}: Error finding container 08637c82cddc68dd2aeea753c7f101927c47080bebf90f8b3299d24c6d52aa31: Status 404 returned error can't find the container with id 08637c82cddc68dd2aeea753c7f101927c47080bebf90f8b3299d24c6d52aa31
Oct 30 22:17:09.678271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:09.675836    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:60028->10.133.0.51:15021: read: connection reset by peer" start-of-body=
Oct 30 22:17:09.678271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:09.675900    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:60028->10.133.0.51:15021: read: connection reset by peer"
Oct 30 22:17:10.005730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.005631    4581 generic.go:356] "Generic (PLEG): container finished" podID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerID="91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f" exitCode=0
Oct 30 22:17:10.005730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.005706    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" event={"ID":"c401bcec-d339-41e0-8a64-ec96d4fdc5eb","Type":"ContainerDied","Data":"91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f"}
Oct 30 22:17:10.006339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.006152    4581 scope.go:122] "RemoveContainer" containerID="91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f"
Oct 30 22:17:10.007810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.007775    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:10.007910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.007861    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:10.007910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.007902    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:10.009268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.009246    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" event={"ID":"46bf79be-6e7c-4780-9d50-5d13d5d57bae","Type":"ContainerStarted","Data":"7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462"}
Oct 30 22:17:10.009355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.009281    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" event={"ID":"46bf79be-6e7c-4780-9d50-5d13d5d57bae","Type":"ContainerStarted","Data":"08637c82cddc68dd2aeea753c7f101927c47080bebf90f8b3299d24c6d52aa31"}
Oct 30 22:17:10.534459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.534416    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:17:10.534649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.534506    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:10.629730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:10.629696    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:10.836582 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:17:10.836470    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:17:11.028061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:11.028020    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" event={"ID":"c401bcec-d339-41e0-8a64-ec96d4fdc5eb","Type":"ContainerStarted","Data":"3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999"}
Oct 30 22:17:11.077851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:11.077781    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podStartSLOduration=3.077757699 podStartE2EDuration="3.077757699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:10.095168057 +0000 UTC m=+614.134106944" watchObservedRunningTime="2026-10-30 22:17:11.077757699 +0000 UTC m=+615.116696586"
Oct 30 22:17:11.261692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:11.261612    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:17:11.266241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:11.264349    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:17:11.266241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:11.264411    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:17:12.260267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:12.259784    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:17:12.260267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:12.259865    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:17:13.274732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:13.274690    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:17:13.284380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:13.275356    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:17:14.057017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:14.055626    4581 generic.go:356] "Generic (PLEG): container finished" podID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerID="171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a" exitCode=0
Oct 30 22:17:14.057017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:14.055735    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" event={"ID":"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5","Type":"ContainerDied","Data":"171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a"}
Oct 30 22:17:14.057017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:14.056201    4581 scope.go:122] "RemoveContainer" containerID="171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a"
Oct 30 22:17:14.059900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:14.059831    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:14.060230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:14.060150    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:14.060445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:14.060399    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:14.282258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:14.260710    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:17:14.282258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:14.260788    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:17:15.105246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:15.099107    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" event={"ID":"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5","Type":"ContainerStarted","Data":"8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3"}
Oct 30 22:17:15.272063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:15.272028    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:17:15.272389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:15.272355    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:17:15.542691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:15.542602    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:17:15.543098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:15.542818    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:17:15.543098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:15.542862    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 22:17:16.270300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:16.257662    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:17:16.270300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:16.257731    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 22:17:16.542700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:16.542595    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:17:16.542700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:16.542658    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 22:17:17.144688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.144654    4581 generic.go:356] "Generic (PLEG): container finished" podID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerID="2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712" exitCode=0
Oct 30 22:17:17.144866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.144782    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" event={"ID":"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175","Type":"ContainerDied","Data":"2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712"}
Oct 30 22:17:17.145193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.145178    4581 scope.go:122] "RemoveContainer" containerID="2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712"
Oct 30 22:17:17.147260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.147188    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:17.147389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.147321    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:17.147389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.147374    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:17:17.282259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.281092    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:17:17.282484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.282450    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:17:17.283319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.283292    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 22:17:17.546304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.543854    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:17:17.546304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.543914    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 22:17:17.712464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.712398    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"]
Oct 30 22:17:17.718080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.717675    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.730130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.730108    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"default-dockercfg-69b2d\""
Oct 30 22:17:17.731379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.730435    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"openshift-service-ca.crt\""
Oct 30 22:17:17.731515 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.730456    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"kube-root-ca.crt\""
Oct 30 22:17:17.731614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.730588    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"simulator-w0-09d0d5-kserve-self-signed-certs\""
Oct 30 22:17:17.744481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.744448    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"]
Oct 30 22:17:17.885258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.884608    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-tmp-dir\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.885258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.884767    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-dshm\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.885258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.884813    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-model-cache\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.885258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.884860    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592923b-e495-48d3-8657-39df0b3db1d9-tls-certs\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.885258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.884994    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6sw\" (UniqueName: \"kubernetes.io/projected/6592923b-e495-48d3-8657-39df0b3db1d9-kube-api-access-ch6sw\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.885258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.885054    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-home\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.986533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.986487    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6sw\" (UniqueName: \"kubernetes.io/projected/6592923b-e495-48d3-8657-39df0b3db1d9-kube-api-access-ch6sw\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.986732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.986558    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-home\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.986732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.986630    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-tmp-dir\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.986732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.986714    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-dshm\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.986867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.986739    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-model-cache\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.986867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.986779    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592923b-e495-48d3-8657-39df0b3db1d9-tls-certs\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.987585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.987523    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-home\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.987585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.987557    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-model-cache\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.987759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.987600    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-tmp-dir\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.990345 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.990325    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-dshm\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:17.990603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:17.990580    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592923b-e495-48d3-8657-39df0b3db1d9-tls-certs\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:18.006241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.006196    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6sw\" (UniqueName: \"kubernetes.io/projected/6592923b-e495-48d3-8657-39df0b3db1d9-kube-api-access-ch6sw\") pod \"simulator-w0-09d0d5-kserve-6f7684b647-62btf\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") " pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:18.034212 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.034173    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:18.158232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.158112    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" event={"ID":"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175","Type":"ContainerStarted","Data":"3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412"}
Oct 30 22:17:18.480507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.480404    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"]
Oct 30 22:17:18.542736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.542704    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:17:18.542902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.542752    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 22:17:18.630659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.630627    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:18.643033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.643008    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:19.001460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.997301    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:17:19.001460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.997453    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:17:19.001460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:18.997527    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:17:19.213614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:19.213578    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" event={"ID":"6592923b-e495-48d3-8657-39df0b3db1d9","Type":"ContainerStarted","Data":"2de2c797ee841d46c9aa28e208cc50a9aee36ccb471c10edc125b04fe76042b9"}
Oct 30 22:17:19.213823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:19.213638    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" event={"ID":"6592923b-e495-48d3-8657-39df0b3db1d9","Type":"ContainerStarted","Data":"c0b44997aab17c0bb2c1c9fd1a13ac3d83ec6327f15cc13d71cddaed5df4d69b"}
Oct 30 22:17:19.225886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:19.225858    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:17:19.249779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:19.249727    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podStartSLOduration=2.249712316 podStartE2EDuration="2.249712316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:19.248187589 +0000 UTC m=+623.287126476" watchObservedRunningTime="2026-10-30 22:17:19.249712316 +0000 UTC m=+623.288651240"
Oct 30 22:17:19.542869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:19.542828    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:17:19.543055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:19.542896    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 22:17:20.002247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:20.001676    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:17:20.002247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:20.001743    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:17:20.034409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:20.034369    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:20.753598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:20.753565    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:17:20.755274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:20.755251    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:17:20.763477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:20.763448    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 22:17:20.994933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:20.994898    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:17:20.995106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:20.994966    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:17:21.249475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.249442    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"]
Oct 30 22:17:21.254302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.254271    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.271296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.269640    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"kube-root-ca.crt\""
Oct 30 22:17:21.284085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.284057    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"default-dockercfg-nnn5d\""
Oct 30 22:17:21.287433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.284274    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"simulator-w5-54114e-kserve-self-signed-certs\""
Oct 30 22:17:21.289157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.289127    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"]
Oct 30 22:17:21.373559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.373525    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"openshift-service-ca.crt\""
Oct 30 22:17:21.374478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.374381    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgl55\" (UniqueName: \"kubernetes.io/projected/24ac387d-d8d6-48f7-abb9-1583f893c4d0-kube-api-access-bgl55\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.374679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.374663    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-model-cache\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.374791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.374773    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tmp-dir\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.374897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.374884    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-dshm\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.375966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.375937    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-home\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.376194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.376176    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tls-certs\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.477337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.477277    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-home\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.477551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.477369    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tls-certs\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.477551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.477428    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgl55\" (UniqueName: \"kubernetes.io/projected/24ac387d-d8d6-48f7-abb9-1583f893c4d0-kube-api-access-bgl55\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.477551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.477459    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-model-cache\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.477551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.477483    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tmp-dir\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.477551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.477507    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-dshm\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.477854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.477792    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-home\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.478247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.478200    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-model-cache\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.478537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.478514    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tmp-dir\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.480599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.480568    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-dshm\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.481360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.481340    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tls-certs\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.510033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.509944    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgl55\" (UniqueName: \"kubernetes.io/projected/24ac387d-d8d6-48f7-abb9-1583f893c4d0-kube-api-access-bgl55\") pod \"simulator-w5-54114e-kserve-7df5c4f9bd-flm8l\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") " pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.596427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.596385    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:21.920982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.920948    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"]
Oct 30 22:17:21.924392 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:17:21.924358    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ac387d_d8d6_48f7_abb9_1583f893c4d0.slice/crio-ed95758d8e5a6f114f033233541cffe656ebeded2bd888ad090d6faaabaf55a0 WatchSource:0}: Error finding container ed95758d8e5a6f114f033233541cffe656ebeded2bd888ad090d6faaabaf55a0: Status 404 returned error can't find the container with id ed95758d8e5a6f114f033233541cffe656ebeded2bd888ad090d6faaabaf55a0
Oct 30 22:17:21.995183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.995147    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:17:21.995619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:21.995592    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:17:22.234870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:22.234777    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" event={"ID":"24ac387d-d8d6-48f7-abb9-1583f893c4d0","Type":"ContainerStarted","Data":"bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7"}
Oct 30 22:17:22.234870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:22.234826    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" event={"ID":"24ac387d-d8d6-48f7-abb9-1583f893c4d0","Type":"ContainerStarted","Data":"ed95758d8e5a6f114f033233541cffe656ebeded2bd888ad090d6faaabaf55a0"}
Oct 30 22:17:22.277880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:22.277809    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podStartSLOduration=1.277787819 podStartE2EDuration="1.277787819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:22.277341811 +0000 UTC m=+626.316280697" watchObservedRunningTime="2026-10-30 22:17:22.277787819 +0000 UTC m=+626.316726708"
Oct 30 22:17:22.596846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:22.596756    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:22.995672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:22.995633    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:17:22.995860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:22.995698    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 22:17:24.008826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:24.008790    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:17:24.009474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:24.009372    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:17:24.010671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:24.010627    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:17:28.035189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.035149    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:28.048018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.047990    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:28.287531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.287444    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:17:28.473326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.473293    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"]
Oct 30 22:17:28.479341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.479309    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.495558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.495532    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"default-dockercfg-pq7cw\""
Oct 30 22:17:28.495558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.495532    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"openshift-service-ca.crt\""
Oct 30 22:17:28.495728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.495605    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"kube-root-ca.crt\""
Oct 30 22:17:28.495728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.495682    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"simulator-w3-d52c59-kserve-self-signed-certs\""
Oct 30 22:17:28.498810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.498778    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"]
Oct 30 22:17:28.575527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.575429    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjrn\" (UniqueName: \"kubernetes.io/projected/4a5cc362-a99e-4622-9e94-a263b53ded58-kube-api-access-mvjrn\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.575527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.575480    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-home\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.575737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.575588    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-tmp-dir\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.575737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.575605    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5cc362-a99e-4622-9e94-a263b53ded58-tls-certs\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.575737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.575631    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-dshm\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.575737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.575725    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-model-cache\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677114    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-tmp-dir\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677148    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5cc362-a99e-4622-9e94-a263b53ded58-tls-certs\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677174    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-dshm\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677278    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-model-cache\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677384    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjrn\" (UniqueName: \"kubernetes.io/projected/4a5cc362-a99e-4622-9e94-a263b53ded58-kube-api-access-mvjrn\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677438    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-home\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677643    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-tmp-dir\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677819    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-model-cache\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.677938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.677852    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-home\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.679772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.679743    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-dshm\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.679961 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.679944    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5cc362-a99e-4622-9e94-a263b53ded58-tls-certs\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.695123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.695099    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjrn\" (UniqueName: \"kubernetes.io/projected/4a5cc362-a99e-4622-9e94-a263b53ded58-kube-api-access-mvjrn\") pod \"simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") " pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:28.792380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:28.792341    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:29.100205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:29.100173    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"]
Oct 30 22:17:29.103289 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:17:29.103258    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a5cc362_a99e_4622_9e94_a263b53ded58.slice/crio-d3f1593e48dc279e4b19f767cade55f72303a8b50899ff3b97a858b3fa6f2dbc WatchSource:0}: Error finding container d3f1593e48dc279e4b19f767cade55f72303a8b50899ff3b97a858b3fa6f2dbc: Status 404 returned error can't find the container with id d3f1593e48dc279e4b19f767cade55f72303a8b50899ff3b97a858b3fa6f2dbc
Oct 30 22:17:29.287295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:29.287256    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" event={"ID":"4a5cc362-a99e-4622-9e94-a263b53ded58","Type":"ContainerStarted","Data":"b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889"}
Oct 30 22:17:29.287500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:29.287309    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" event={"ID":"4a5cc362-a99e-4622-9e94-a263b53ded58","Type":"ContainerStarted","Data":"d3f1593e48dc279e4b19f767cade55f72303a8b50899ff3b97a858b3fa6f2dbc"}
Oct 30 22:17:29.322939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:29.322884    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podStartSLOduration=1.322871723 podStartE2EDuration="1.322871723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:29.322068867 +0000 UTC m=+633.361007752" watchObservedRunningTime="2026-10-30 22:17:29.322871723 +0000 UTC m=+633.361810608"
Oct 30 22:17:29.792793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:29.792736    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:31.597100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:31.597058    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:31.609720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:31.609689    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:32.320709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:32.320670    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:17:37.728770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.728724    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"]
Oct 30 22:17:37.732960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.732940    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.754553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.754498    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"default-dockercfg-2wmx7\""
Oct 30 22:17:37.754730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.754553    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"simulator-w6-012ba0-kserve-self-signed-certs\""
Oct 30 22:17:37.754981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.754964    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"kube-root-ca.crt\""
Oct 30 22:17:37.758427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.758406    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"openshift-service-ca.crt\""
Oct 30 22:17:37.758636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.758615    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"]
Oct 30 22:17:37.884379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.884331    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-dshm\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.884573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.884402    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e0a99-1b6c-4077-92a6-6182d6c63150-tls-certs\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.884573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.884489    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-model-cache\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.884573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.884546    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-tmp-dir\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.884712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.884642    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgklh\" (UniqueName: \"kubernetes.io/projected/058e0a99-1b6c-4077-92a6-6182d6c63150-kube-api-access-sgklh\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.884755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.884736    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-home\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.989017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.988856    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgklh\" (UniqueName: \"kubernetes.io/projected/058e0a99-1b6c-4077-92a6-6182d6c63150-kube-api-access-sgklh\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.989017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.988958    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-home\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.989290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.989058    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-dshm\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.989290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.989110    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e0a99-1b6c-4077-92a6-6182d6c63150-tls-certs\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.989290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.989150    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-model-cache\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.989290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.989201    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-tmp-dir\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.989853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.989825    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-tmp-dir\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.990735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.990706    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-model-cache\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.990916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.990870    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-home\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.992726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.992703    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-dshm\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:37.994028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:37.994006    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e0a99-1b6c-4077-92a6-6182d6c63150-tls-certs\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:38.009656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:38.009625    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgklh\" (UniqueName: \"kubernetes.io/projected/058e0a99-1b6c-4077-92a6-6182d6c63150-kube-api-access-sgklh\") pod \"simulator-w6-012ba0-kserve-d8f4d985f-97qgl\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") " pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:38.048986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:38.048958    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:38.352187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:38.352108    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"]
Oct 30 22:17:38.355764 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:17:38.355735    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod058e0a99_1b6c_4077_92a6_6182d6c63150.slice/crio-5d235d0ecdba9c16730b7f03e32cf18c93191c52a1a8e8b45c94e137cbd2bca5 WatchSource:0}: Error finding container 5d235d0ecdba9c16730b7f03e32cf18c93191c52a1a8e8b45c94e137cbd2bca5: Status 404 returned error can't find the container with id 5d235d0ecdba9c16730b7f03e32cf18c93191c52a1a8e8b45c94e137cbd2bca5
Oct 30 22:17:38.793525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:38.793480    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:38.806239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:38.806191    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:39.356608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:39.356569    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" event={"ID":"058e0a99-1b6c-4077-92a6-6182d6c63150","Type":"ContainerStarted","Data":"a36607d55aebe1e7169391ccac3d40419441f5738a7c00719ff64cef0b328398"}
Oct 30 22:17:39.356808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:39.356618    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" event={"ID":"058e0a99-1b6c-4077-92a6-6182d6c63150","Type":"ContainerStarted","Data":"5d235d0ecdba9c16730b7f03e32cf18c93191c52a1a8e8b45c94e137cbd2bca5"}
Oct 30 22:17:39.368339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:39.368311    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:17:39.395692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:39.395635    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podStartSLOduration=2.395620609 podStartE2EDuration="2.395620609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:39.392478885 +0000 UTC m=+643.431417771" watchObservedRunningTime="2026-10-30 22:17:39.395620609 +0000 UTC m=+643.434559495"
Oct 30 22:17:40.049930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:40.049899    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:40.816096 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:17:40.816060    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:17:45.987144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:45.987104    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"]
Oct 30 22:17:45.991459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:45.991433    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.013508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.013480    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"premium-w2-421583-kserve-self-signed-certs\""
Oct 30 22:17:46.017177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.017148    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"]
Oct 30 22:17:46.085128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.085097    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-home\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.085335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.085152    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-model-cache\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.085335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.085253    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-dshm\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.085335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.085283    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156793bd-b877-497a-8c48-7a1f982a5e78-tls-certs\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.085335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.085301    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-tmp-dir\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.085496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.085437    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plt6r\" (UniqueName: \"kubernetes.io/projected/156793bd-b877-497a-8c48-7a1f982a5e78-kube-api-access-plt6r\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.189231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.189172    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-model-cache\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.190113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.190087    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-dshm\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.190711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.190690    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156793bd-b877-497a-8c48-7a1f982a5e78-tls-certs\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.193490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.193465    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-tmp-dir\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.193767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.193745    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plt6r\" (UniqueName: \"kubernetes.io/projected/156793bd-b877-497a-8c48-7a1f982a5e78-kube-api-access-plt6r\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.194000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.193974    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-tmp-dir\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.194534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.194002    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-home\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.194757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.189996    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-model-cache\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.194757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.194292    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-home\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.200938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.200908    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-dshm\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.201705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.201646    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156793bd-b877-497a-8c48-7a1f982a5e78-tls-certs\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.211551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.211522    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plt6r\" (UniqueName: \"kubernetes.io/projected/156793bd-b877-497a-8c48-7a1f982a5e78-kube-api-access-plt6r\") pod \"premium-w2-421583-kserve-655b46c7dc-hkdcq\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") " pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.305471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.305366    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:46.612077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:46.612035    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"]
Oct 30 22:17:46.622383 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:17:46.622356    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod156793bd_b877_497a_8c48_7a1f982a5e78.slice/crio-24b6bf99c3be18fc542c832499f5dafac3f05d52fa5ac83440800845264c618a WatchSource:0}: Error finding container 24b6bf99c3be18fc542c832499f5dafac3f05d52fa5ac83440800845264c618a: Status 404 returned error can't find the container with id 24b6bf99c3be18fc542c832499f5dafac3f05d52fa5ac83440800845264c618a
Oct 30 22:17:47.422382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:47.422342    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" event={"ID":"156793bd-b877-497a-8c48-7a1f982a5e78","Type":"ContainerStarted","Data":"cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb"}
Oct 30 22:17:47.422810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:47.422394    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" event={"ID":"156793bd-b877-497a-8c48-7a1f982a5e78","Type":"ContainerStarted","Data":"24b6bf99c3be18fc542c832499f5dafac3f05d52fa5ac83440800845264c618a"}
Oct 30 22:17:47.459933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:47.459881    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podStartSLOduration=2.45986507 podStartE2EDuration="2.45986507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:47.456468849 +0000 UTC m=+651.495407726" watchObservedRunningTime="2026-10-30 22:17:47.45986507 +0000 UTC m=+651.498803955"
Oct 30 22:17:48.050069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:48.050030    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:48.063007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:48.062969    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:48.306209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:48.306119    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:48.439577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:48.439548    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:17:56.305808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:56.305760    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:56.319011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:56.318983    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:17:56.498490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:17:56.498452    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:18:06.347393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.347295    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"]
Oct 30 22:18:06.353410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.353382    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.377966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.377935    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"premium-w3-d52c59-kserve-self-signed-certs\""
Oct 30 22:18:06.382875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.382843    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"]
Oct 30 22:18:06.433978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.433929    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-model-cache\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.433978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.433981    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tls-certs\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.434299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.434075    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-home\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.434299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.434197    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd9zp\" (UniqueName: \"kubernetes.io/projected/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-kube-api-access-pd9zp\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.434400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.434305    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-dshm\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.434400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.434353    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tmp-dir\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.535269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.535200    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd9zp\" (UniqueName: \"kubernetes.io/projected/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-kube-api-access-pd9zp\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.535477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.535295    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-dshm\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.535477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.535342    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tmp-dir\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.535606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.535506    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-model-cache\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.535606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.535532    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tls-certs\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.535606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.535578    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-home\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.535986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.535960    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-home\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.536149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.536071    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-model-cache\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.536853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.536830    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tmp-dir\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.540881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.540835    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-dshm\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.545798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.545674    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tls-certs\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.561310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.561205    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd9zp\" (UniqueName: \"kubernetes.io/projected/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-kube-api-access-pd9zp\") pod \"premium-w3-d52c59-kserve-556b5fc797-mkncg\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") " pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.671176 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.671119    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:06.971633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:06.971545    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"]
Oct 30 22:18:06.974827 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:18:06.974783    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fef4a56_7d62_4fc7_b3f8_1d372b71fe08.slice/crio-b6be8b12a3a2b21d1b96c24f4d684e0a04b329c71f53e8b38fbf047b3bbc9c2c WatchSource:0}: Error finding container b6be8b12a3a2b21d1b96c24f4d684e0a04b329c71f53e8b38fbf047b3bbc9c2c: Status 404 returned error can't find the container with id b6be8b12a3a2b21d1b96c24f4d684e0a04b329c71f53e8b38fbf047b3bbc9c2c
Oct 30 22:18:07.565155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:07.565118    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" event={"ID":"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08","Type":"ContainerStarted","Data":"313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8"}
Oct 30 22:18:07.565564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:07.565162    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" event={"ID":"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08","Type":"ContainerStarted","Data":"b6be8b12a3a2b21d1b96c24f4d684e0a04b329c71f53e8b38fbf047b3bbc9c2c"}
Oct 30 22:18:07.599210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:07.599151    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podStartSLOduration=1.599137105 podStartE2EDuration="1.599137105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:07.597518118 +0000 UTC m=+671.636457004" watchObservedRunningTime="2026-10-30 22:18:07.599137105 +0000 UTC m=+671.638075991"
Oct 30 22:18:07.672061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:07.672023    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:10.817718 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:18:10.817674    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:18:16.672154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:16.672114    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:16.690711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:16.690680    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:17.654766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:17.654732    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:18:40.820362 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:18:40.820323    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:18:43.555690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.555642    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"]
Oct 30 22:18:43.561756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.561727    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.580898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.580862    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"]
Oct 30 22:18:43.586684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.586048    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:18:43.586684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.586076    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:18:43.586684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.586094    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:18:43.586684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.586100    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:43.731467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.731432    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbxk\" (UniqueName: \"kubernetes.io/projected/78f705f4-0efd-4492-a198-c866ee875422-kube-api-access-tvbxk\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.731650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.731479    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78f705f4-0efd-4492-a198-c866ee875422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.731650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.731499    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78f705f4-0efd-4492-a198-c866ee875422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.832745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.832662    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbxk\" (UniqueName: \"kubernetes.io/projected/78f705f4-0efd-4492-a198-c866ee875422-kube-api-access-tvbxk\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.832745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.832714    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78f705f4-0efd-4492-a198-c866ee875422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.832948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.832745    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78f705f4-0efd-4492-a198-c866ee875422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.833485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.833457    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78f705f4-0efd-4492-a198-c866ee875422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.835322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.835299    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78f705f4-0efd-4492-a198-c866ee875422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.850128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.850092    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbxk\" (UniqueName: \"kubernetes.io/projected/78f705f4-0efd-4492-a198-c866ee875422-kube-api-access-tvbxk\") pod \"maas-api-revoke-sub-dcbfd6ff-zrvt7\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:43.878682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:43.878649    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:44.191125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:44.191100    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"]
Oct 30 22:18:44.194947 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:18:44.194901    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f705f4_0efd_4492_a198_c866ee875422.slice/crio-1f3a32f269a4932a1d953fdcb53f14dc58854265b1a150a31d913165390de4a3 WatchSource:0}: Error finding container 1f3a32f269a4932a1d953fdcb53f14dc58854265b1a150a31d913165390de4a3: Status 404 returned error can't find the container with id 1f3a32f269a4932a1d953fdcb53f14dc58854265b1a150a31d913165390de4a3
Oct 30 22:18:44.859590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:44.859554    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7" event={"ID":"78f705f4-0efd-4492-a198-c866ee875422","Type":"ContainerStarted","Data":"983e316fd1f950e81a669252adbc0db3fa4a51f266a10a76cc7faf967d7f7584"}
Oct 30 22:18:44.860114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:44.859613    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7" event={"ID":"78f705f4-0efd-4492-a198-c866ee875422","Type":"ContainerStarted","Data":"1f3a32f269a4932a1d953fdcb53f14dc58854265b1a150a31d913165390de4a3"}
Oct 30 22:18:44.896100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:44.896036    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7" podStartSLOduration=1.8960163300000001 podStartE2EDuration="1.89601633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:44.894131436 +0000 UTC m=+708.933070323" watchObservedRunningTime="2026-10-30 22:18:44.89601633 +0000 UTC m=+708.934955221"
Oct 30 22:18:45.885394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:45.882341    4581 generic.go:356] "Generic (PLEG): container finished" podID="78f705f4-0efd-4492-a198-c866ee875422" containerID="983e316fd1f950e81a669252adbc0db3fa4a51f266a10a76cc7faf967d7f7584" exitCode=0
Oct 30 22:18:45.885394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:45.882532    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7" event={"ID":"78f705f4-0efd-4492-a198-c866ee875422","Type":"ContainerDied","Data":"983e316fd1f950e81a669252adbc0db3fa4a51f266a10a76cc7faf967d7f7584"}
Oct 30 22:18:47.068850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.068820    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:47.170774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.170671    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78f705f4-0efd-4492-a198-c866ee875422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78f705f4-0efd-4492-a198-c866ee875422-maas-api-openshift-ca\") pod \"78f705f4-0efd-4492-a198-c866ee875422\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") "
Oct 30 22:18:47.170936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.170791    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78f705f4-0efd-4492-a198-c866ee875422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78f705f4-0efd-4492-a198-c866ee875422-maas-api-xks-ca\") pod \"78f705f4-0efd-4492-a198-c866ee875422\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") "
Oct 30 22:18:47.170936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.170839    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78f705f4-0efd-4492-a198-c866ee875422-kube-api-access-tvbxk\" (UniqueName: \"kubernetes.io/projected/78f705f4-0efd-4492-a198-c866ee875422-kube-api-access-tvbxk\") pod \"78f705f4-0efd-4492-a198-c866ee875422\" (UID: \"78f705f4-0efd-4492-a198-c866ee875422\") "
Oct 30 22:18:47.172525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.171344    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f705f4-0efd-4492-a198-c866ee875422-maas-api-openshift-ca" pod "78f705f4-0efd-4492-a198-c866ee875422" (UID: "78f705f4-0efd-4492-a198-c866ee875422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:47.172525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.171510    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78f705f4-0efd-4492-a198-c866ee875422-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:18:47.174085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.173511    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f705f4-0efd-4492-a198-c866ee875422-maas-api-xks-ca" pod "78f705f4-0efd-4492-a198-c866ee875422" (UID: "78f705f4-0efd-4492-a198-c866ee875422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:47.174085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.173657    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f705f4-0efd-4492-a198-c866ee875422-kube-api-access-tvbxk" pod "78f705f4-0efd-4492-a198-c866ee875422" (UID: "78f705f4-0efd-4492-a198-c866ee875422"). InnerVolumeSpecName "kube-api-access-tvbxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:47.272632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.272597    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78f705f4-0efd-4492-a198-c866ee875422-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:18:47.272632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.272627    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbxk\" (UniqueName: \"kubernetes.io/projected/78f705f4-0efd-4492-a198-c866ee875422-kube-api-access-tvbxk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:18:47.904446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.904408    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7" event={"ID":"78f705f4-0efd-4492-a198-c866ee875422","Type":"ContainerDied","Data":"1f3a32f269a4932a1d953fdcb53f14dc58854265b1a150a31d913165390de4a3"}
Oct 30 22:18:47.904620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.904467    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f3a32f269a4932a1d953fdcb53f14dc58854265b1a150a31d913165390de4a3"
Oct 30 22:18:47.904620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:47.904540    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"
Oct 30 22:18:52.189436 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.189395    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"]
Oct 30 22:18:52.190150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.190126    4581 cpu_manager.go:414] "containerMap: removing container" podUID="78f705f4-0efd-4492-a198-c866ee875422" containerName="revoke-keys"
Oct 30 22:18:52.190267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.190152    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f705f4-0efd-4492-a198-c866ee875422" containerName="revoke-keys"
Oct 30 22:18:52.190344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.190330    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f705f4-0efd-4492-a198-c866ee875422" containerName="revoke-keys"
Oct 30 22:18:52.194800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.194773    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.207082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.207052    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d52c59-openshift-default-dockercfg-x7nqp\""
Oct 30 22:18:52.222640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.222605    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"]
Oct 30 22:18:52.333101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333066    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.333101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333105    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9tr\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-kube-api-access-kf9tr\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.333388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333133    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.333388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333284    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.333388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333350    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.333524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333418    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.333524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333488    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.333597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333540    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae9826f-c2f9-485f-82e3-179c3493ff78-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.333597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.333562    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434479    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434533    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434588    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae9826f-c2f9-485f-82e3-179c3493ff78-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434618    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434722    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434751    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9tr\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-kube-api-access-kf9tr\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434791    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434839    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.434953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.434875    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.435146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.435066    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.435510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.435476    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae9826f-c2f9-485f-82e3-179c3493ff78-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.435510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.435477    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.435649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.435611    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.435712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.435689    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.437252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.437195    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.437939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.437922    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.451535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.451472    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.451535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.451522    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9tr\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-kube-api-access-kf9tr\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.509814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.509780    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:52.835559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.835476    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"]
Oct 30 22:18:52.839418 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:18:52.839366    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae9826f_c2f9_485f_82e3_179c3493ff78.slice/crio-ee9d3bd696d090443eab60a60f7ee784c728ccd1450a146292c95d3ddb4945df WatchSource:0}: Error finding container ee9d3bd696d090443eab60a60f7ee784c728ccd1450a146292c95d3ddb4945df: Status 404 returned error can't find the container with id ee9d3bd696d090443eab60a60f7ee784c728ccd1450a146292c95d3ddb4945df
Oct 30 22:18:52.841966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.841924    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:52.842060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.842018    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:52.842121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.842093    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:18:52.938639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.938600    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" event={"ID":"eae9826f-c2f9-485f-82e3-179c3493ff78","Type":"ContainerStarted","Data":"bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c"}
Oct 30 22:18:52.938795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.938666    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" event={"ID":"eae9826f-c2f9-485f-82e3-179c3493ff78","Type":"ContainerStarted","Data":"ee9d3bd696d090443eab60a60f7ee784c728ccd1450a146292c95d3ddb4945df"}
Oct 30 22:18:52.975344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:52.975282    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podStartSLOduration=0.975268473 podStartE2EDuration="975.268473ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:52.974746663 +0000 UTC m=+717.013685551" watchObservedRunningTime="2026-10-30 22:18:52.975268473 +0000 UTC m=+717.014207387"
Oct 30 22:18:53.511700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:53.510320    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:18:53.512142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:53.511724    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:18:53.512142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:53.511774    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:18:54.511261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:54.511199    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:18:54.511450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:54.511288    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:18:55.510266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:55.510207    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:18:55.510677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:55.510298    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:18:56.510842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:56.510801    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:18:56.511305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:56.510868    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:18:57.511281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.511211    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:18:57.511689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.511319    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:18:57.893286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.893252    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"]
Oct 30 22:18:57.897854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.897826    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:57.912832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.912804    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:57.932117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.932063    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"]
Oct 30 22:18:57.932575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.932537    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:18:57.932778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.932766    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:18:57.935364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:57.935342    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:18:58.019940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.019909    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.020113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.020044    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.020157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.020112    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtcq\" (UniqueName: \"kubernetes.io/projected/8113895f-1e4e-4742-84b7-2114231170bd-kube-api-access-hjtcq\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.121494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.121464    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.121693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.121531    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtcq\" (UniqueName: \"kubernetes.io/projected/8113895f-1e4e-4742-84b7-2114231170bd-kube-api-access-hjtcq\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.121693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.121595    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.122279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.122260    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.124095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.124063    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.139739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.139713    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtcq\" (UniqueName: \"kubernetes.io/projected/8113895f-1e4e-4742-84b7-2114231170bd-kube-api-access-hjtcq\") pod \"maas-api-revoke-sub-786bd150-xz9vj\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.217846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.217760    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:18:58.510535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.510441    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:18:58.510535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.510498    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:18:58.588236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:58.588178    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"]
Oct 30 22:18:58.591681 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:18:58.591650    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8113895f_1e4e_4742_84b7_2114231170bd.slice/crio-d2db7c60a20e017bbcc67989d9f27c3701ca89706a626f3e28831179592dc77d WatchSource:0}: Error finding container d2db7c60a20e017bbcc67989d9f27c3701ca89706a626f3e28831179592dc77d: Status 404 returned error can't find the container with id d2db7c60a20e017bbcc67989d9f27c3701ca89706a626f3e28831179592dc77d
Oct 30 22:18:59.004414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.004376    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj" event={"ID":"8113895f-1e4e-4742-84b7-2114231170bd","Type":"ContainerStarted","Data":"8b25fd899688b0bd0237b2958549e0925fca012e30cce7a23fbe55f3772a4318"}
Oct 30 22:18:59.004613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.004437    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj" event={"ID":"8113895f-1e4e-4742-84b7-2114231170bd","Type":"ContainerStarted","Data":"d2db7c60a20e017bbcc67989d9f27c3701ca89706a626f3e28831179592dc77d"}
Oct 30 22:18:59.040834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.040778    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj" podStartSLOduration=2.040763682 podStartE2EDuration="2.040763682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:59.039504448 +0000 UTC m=+723.078443333" watchObservedRunningTime="2026-10-30 22:18:59.040763682 +0000 UTC m=+723.079702567"
Oct 30 22:18:59.068590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.068547    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"]
Oct 30 22:18:59.073444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.073417    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.094872 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.094832    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"]
Oct 30 22:18:59.233968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.233870    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.233968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.233914    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5c2q\" (UniqueName: \"kubernetes.io/projected/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-kube-api-access-v5c2q\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.234204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.234107    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.335484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.335449    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.335484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.335490    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5c2q\" (UniqueName: \"kubernetes.io/projected/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-kube-api-access-v5c2q\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.335737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.335613    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.336143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.336121    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.338175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.338148    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.353020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.352988    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5c2q\" (UniqueName: \"kubernetes.io/projected/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-kube-api-access-v5c2q\") pod \"maas-api-revoke-sub-1cd8311b-fxzht\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.389139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.389094    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:18:59.512023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.510639    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:18:59.512023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.510706    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 22:18:59.703628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:18:59.703586    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"]
Oct 30 22:18:59.707800 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:18:59.707770    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9562d9c5_c3b4_4ccf_906a_f1be94ba501e.slice/crio-50c70460c0358fef7e7a78dd4941e9c6e60d9e1f45851a33990c250e545a6337 WatchSource:0}: Error finding container 50c70460c0358fef7e7a78dd4941e9c6e60d9e1f45851a33990c250e545a6337: Status 404 returned error can't find the container with id 50c70460c0358fef7e7a78dd4941e9c6e60d9e1f45851a33990c250e545a6337
Oct 30 22:19:00.012963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:00.012926    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht" event={"ID":"9562d9c5-c3b4-4ccf-906a-f1be94ba501e","Type":"ContainerStarted","Data":"b16170679b146db1c43477b03c87ee33d9b6276a337e0bec876f8e689c889f77"}
Oct 30 22:19:00.013156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:00.012988    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht" event={"ID":"9562d9c5-c3b4-4ccf-906a-f1be94ba501e","Type":"ContainerStarted","Data":"50c70460c0358fef7e7a78dd4941e9c6e60d9e1f45851a33990c250e545a6337"}
Oct 30 22:19:00.014601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:00.014575    4581 generic.go:356] "Generic (PLEG): container finished" podID="8113895f-1e4e-4742-84b7-2114231170bd" containerID="8b25fd899688b0bd0237b2958549e0925fca012e30cce7a23fbe55f3772a4318" exitCode=0
Oct 30 22:19:00.014723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:00.014659    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj" event={"ID":"8113895f-1e4e-4742-84b7-2114231170bd","Type":"ContainerDied","Data":"8b25fd899688b0bd0237b2958549e0925fca012e30cce7a23fbe55f3772a4318"}
Oct 30 22:19:00.060999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:00.060943    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht" podStartSLOduration=1.060923313 podStartE2EDuration="1.060923313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:00.04986905 +0000 UTC m=+724.088807935" watchObservedRunningTime="2026-10-30 22:19:00.060923313 +0000 UTC m=+724.099862203"
Oct 30 22:19:00.515175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:00.515079    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:19:00.515518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:00.515497    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:19:00.516178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:00.516155    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:19:01.022999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.022964    4581 generic.go:356] "Generic (PLEG): container finished" podID="9562d9c5-c3b4-4ccf-906a-f1be94ba501e" containerID="b16170679b146db1c43477b03c87ee33d9b6276a337e0bec876f8e689c889f77" exitCode=0
Oct 30 22:19:01.023464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.023050    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht" event={"ID":"9562d9c5-c3b4-4ccf-906a-f1be94ba501e","Type":"ContainerDied","Data":"b16170679b146db1c43477b03c87ee33d9b6276a337e0bec876f8e689c889f77"}
Oct 30 22:19:01.204498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.204469    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:19:01.362772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.362740    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-xks-ca\") pod \"8113895f-1e4e-4742-84b7-2114231170bd\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") "
Oct 30 22:19:01.362965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.362896    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8113895f-1e4e-4742-84b7-2114231170bd-kube-api-access-hjtcq\" (UniqueName: \"kubernetes.io/projected/8113895f-1e4e-4742-84b7-2114231170bd-kube-api-access-hjtcq\") pod \"8113895f-1e4e-4742-84b7-2114231170bd\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") "
Oct 30 22:19:01.363055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.363036    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-openshift-ca\") pod \"8113895f-1e4e-4742-84b7-2114231170bd\" (UID: \"8113895f-1e4e-4742-84b7-2114231170bd\") "
Oct 30 22:19:01.363429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.363402    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-openshift-ca" pod "8113895f-1e4e-4742-84b7-2114231170bd" (UID: "8113895f-1e4e-4742-84b7-2114231170bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:01.363577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.363555    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:01.365305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.365276    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8113895f-1e4e-4742-84b7-2114231170bd-kube-api-access-hjtcq" pod "8113895f-1e4e-4742-84b7-2114231170bd" (UID: "8113895f-1e4e-4742-84b7-2114231170bd"). InnerVolumeSpecName "kube-api-access-hjtcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:01.365491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.365472    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-xks-ca" pod "8113895f-1e4e-4742-84b7-2114231170bd" (UID: "8113895f-1e4e-4742-84b7-2114231170bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:01.464636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.464594    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjtcq\" (UniqueName: \"kubernetes.io/projected/8113895f-1e4e-4742-84b7-2114231170bd-kube-api-access-hjtcq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:01.464636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:01.464638    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113895f-1e4e-4742-84b7-2114231170bd-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:02.031723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.031691    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"
Oct 30 22:19:02.032164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.031723    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj" event={"ID":"8113895f-1e4e-4742-84b7-2114231170bd","Type":"ContainerDied","Data":"d2db7c60a20e017bbcc67989d9f27c3701ca89706a626f3e28831179592dc77d"}
Oct 30 22:19:02.032164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.031769    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2db7c60a20e017bbcc67989d9f27c3701ca89706a626f3e28831179592dc77d"
Oct 30 22:19:02.212241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.207643    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:19:02.382099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.382053    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-kube-api-access-v5c2q\" (UniqueName: \"kubernetes.io/projected/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-kube-api-access-v5c2q\") pod \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") "
Oct 30 22:19:02.382365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.382110    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-openshift-ca\") pod \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") "
Oct 30 22:19:02.382631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.382599    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-openshift-ca" pod "9562d9c5-c3b4-4ccf-906a-f1be94ba501e" (UID: "9562d9c5-c3b4-4ccf-906a-f1be94ba501e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:02.382834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.382815    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-xks-ca\") pod \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\" (UID: \"9562d9c5-c3b4-4ccf-906a-f1be94ba501e\") "
Oct 30 22:19:02.383269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.383245    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:02.396243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.386172    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-xks-ca" pod "9562d9c5-c3b4-4ccf-906a-f1be94ba501e" (UID: "9562d9c5-c3b4-4ccf-906a-f1be94ba501e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:02.402587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.402390    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-kube-api-access-v5c2q" pod "9562d9c5-c3b4-4ccf-906a-f1be94ba501e" (UID: "9562d9c5-c3b4-4ccf-906a-f1be94ba501e"). InnerVolumeSpecName "kube-api-access-v5c2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:02.483925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.483881    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:02.483925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:02.483917    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5c2q\" (UniqueName: \"kubernetes.io/projected/9562d9c5-c3b4-4ccf-906a-f1be94ba501e-kube-api-access-v5c2q\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:03.043021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:03.042980    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht" event={"ID":"9562d9c5-c3b4-4ccf-906a-f1be94ba501e","Type":"ContainerDied","Data":"50c70460c0358fef7e7a78dd4941e9c6e60d9e1f45851a33990c250e545a6337"}
Oct 30 22:19:03.043021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:03.043014    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"
Oct 30 22:19:03.043570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:03.043070    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c70460c0358fef7e7a78dd4941e9c6e60d9e1f45851a33990c250e545a6337"
Oct 30 22:19:07.504458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.504424    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"]
Oct 30 22:19:07.504979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.504966    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8113895f-1e4e-4742-84b7-2114231170bd" containerName="revoke-keys"
Oct 30 22:19:07.505018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.504981    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8113895f-1e4e-4742-84b7-2114231170bd" containerName="revoke-keys"
Oct 30 22:19:07.505018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.504992    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9562d9c5-c3b4-4ccf-906a-f1be94ba501e" containerName="revoke-keys"
Oct 30 22:19:07.505018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.504997    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9562d9c5-c3b4-4ccf-906a-f1be94ba501e" containerName="revoke-keys"
Oct 30 22:19:07.505126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.505087    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9562d9c5-c3b4-4ccf-906a-f1be94ba501e" containerName="revoke-keys"
Oct 30 22:19:07.505126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.505102    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8113895f-1e4e-4742-84b7-2114231170bd" containerName="revoke-keys"
Oct 30 22:19:07.508642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.508621    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.523918 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.523889    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0ad7c5-openshift-default-dockercfg-b9j97\""
Oct 30 22:19:07.544006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.543957    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"]
Oct 30 22:19:07.559962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.559919    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86ad25d-cba7-4736-add7-cc1865fe0feb-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.560162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.559979    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.560162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.560015    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.560162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.560078    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.560162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.560124    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.560433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.560293    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhm5k\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-kube-api-access-nhm5k\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.560433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.560330    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.560433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.560374    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.560433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.560414    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661206    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661301    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661432    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm5k\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-kube-api-access-nhm5k\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661458    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661498    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661543    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661588    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86ad25d-cba7-4736-add7-cc1865fe0feb-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661613    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661641    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.661856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661646    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.662014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.661995    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.662268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.662232    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.662440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.662420    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86ad25d-cba7-4736-add7-cc1865fe0feb-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.662508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.662468    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.664307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.664274    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.664428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.664401    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.679912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.679881    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhm5k\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-kube-api-access-nhm5k\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.680169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.680144    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:07.822821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:07.822726    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:08.193983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:08.192133    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"]
Oct 30 22:19:08.200241 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:08.197616    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86ad25d_cba7_4736_add7_cc1865fe0feb.slice/crio-10f9dfe91e44b9ab39aea09c68908178a0e764512acbdade43ecb12c6609fc36 WatchSource:0}: Error finding container 10f9dfe91e44b9ab39aea09c68908178a0e764512acbdade43ecb12c6609fc36: Status 404 returned error can't find the container with id 10f9dfe91e44b9ab39aea09c68908178a0e764512acbdade43ecb12c6609fc36
Oct 30 22:19:08.200241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:08.200177    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:08.200551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:08.200271    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:08.200551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:08.200311    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:09.122497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:09.122450    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" event={"ID":"c86ad25d-cba7-4736-add7-cc1865fe0feb","Type":"ContainerStarted","Data":"1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2"}
Oct 30 22:19:09.122984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:09.122528    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" event={"ID":"c86ad25d-cba7-4736-add7-cc1865fe0feb","Type":"ContainerStarted","Data":"10f9dfe91e44b9ab39aea09c68908178a0e764512acbdade43ecb12c6609fc36"}
Oct 30 22:19:09.180246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:09.176168    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podStartSLOduration=2.176143386 podStartE2EDuration="2.176143386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:09.159900744 +0000 UTC m=+733.198839630" watchObservedRunningTime="2026-10-30 22:19:09.176143386 +0000 UTC m=+733.215082273"
Oct 30 22:19:09.823913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:09.823870    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:09.825517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:09.825478    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:19:09.825632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:09.825534    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:19:10.823731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:10.823694    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:19:10.824243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:10.823752    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:19:10.828763 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:19:10.828732    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:19:11.478868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.478819    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"]
Oct 30 22:19:11.484423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.484395    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.488978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.488947    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"]
Oct 30 22:19:11.496403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.496382    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:11.506279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.506259    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:11.506425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.506262    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:11.506482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.506207    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:19:11.534997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.534957    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.535379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.535354    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdz5\" (UniqueName: \"kubernetes.io/projected/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-kube-api-access-rsdz5\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.535549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.535533    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.637028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.636990    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdz5\" (UniqueName: \"kubernetes.io/projected/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-kube-api-access-rsdz5\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.637260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.637046    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.637260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.637134    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.637846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.637818    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.639761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.639740    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.683019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.682986    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdz5\" (UniqueName: \"kubernetes.io/projected/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-kube-api-access-rsdz5\") pod \"maas-api-revoke-sub-1d86710a-nbwlh\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.803156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.803074    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:11.824087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.824046    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:19:11.824469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:11.824128    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:19:12.145569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:12.142047    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"]
Oct 30 22:19:12.826783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:12.826686    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:19:12.827437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:12.827403    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:19:13.193864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.193827    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh" event={"ID":"dafbf2ee-8c6d-4897-a701-6f90ad2f3422","Type":"ContainerStarted","Data":"c6f8d87b4c2cb04684d44b9fb4d1a87ec2a2bf69d17a8aefcb7e75252442a096"}
Oct 30 22:19:13.194158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.194127    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh" event={"ID":"dafbf2ee-8c6d-4897-a701-6f90ad2f3422","Type":"ContainerStarted","Data":"371b6a0cd884f994c755753bf8f01adc314237832e1c3e22043dcad4ffef579c"}
Oct 30 22:19:13.230919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.230867    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh" podStartSLOduration=2.230852937 podStartE2EDuration="2.230852937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:13.2269553 +0000 UTC m=+737.265894187" watchObservedRunningTime="2026-10-30 22:19:13.230852937 +0000 UTC m=+737.269791823"
Oct 30 22:19:13.531601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.531519    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"]
Oct 30 22:19:13.535460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.535432    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.601973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.594691    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"]
Oct 30 22:19:13.601973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.594908    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xjp\" (UniqueName: \"kubernetes.io/projected/d74c0295-9424-462a-95d9-7170777ed5e2-kube-api-access-24xjp\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.601973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.594978    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.601973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.595037    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.669244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.659851    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"]
Oct 30 22:19:13.669244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.664983    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.692283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.692251    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"]
Oct 30 22:19:13.696712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.696675    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xjp\" (UniqueName: \"kubernetes.io/projected/d74c0295-9424-462a-95d9-7170777ed5e2-kube-api-access-24xjp\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.697395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.697364    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.700294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.697427    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.700294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.699359    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.703013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.702982    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.713776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.713752    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xjp\" (UniqueName: \"kubernetes.io/projected/d74c0295-9424-462a-95d9-7170777ed5e2-kube-api-access-24xjp\") pod \"maas-api-revoke-sub-716b7868-m22hj\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.798845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.798736    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.798845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.798786    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.799081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.798873    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcc4v\" (UniqueName: \"kubernetes.io/projected/a9a2e504-9393-4da1-bd67-4b58fa4510a2-kube-api-access-lcc4v\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.823725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.823686    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:19:13.823884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.823741    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:19:13.848255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.848203    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:13.901474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.901434    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.901617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.901494    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.901617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.901555    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcc4v\" (UniqueName: \"kubernetes.io/projected/a9a2e504-9393-4da1-bd67-4b58fa4510a2-kube-api-access-lcc4v\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.902653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.902598    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.904700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.904677    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:13.918544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:13.918523    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcc4v\" (UniqueName: \"kubernetes.io/projected/a9a2e504-9393-4da1-bd67-4b58fa4510a2-kube-api-access-lcc4v\") pod \"maas-api-revoke-sub-f8c82a37-lxmqh\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:14.001644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:14.001600    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:14.196052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:14.195869    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"]
Oct 30 22:19:14.201164 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:14.201127    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74c0295_9424_462a_95d9_7170777ed5e2.slice/crio-6f30c2cc6cb79dc28619a9373d69b4cb003517686d057b5487101526efc8ec88 WatchSource:0}: Error finding container 6f30c2cc6cb79dc28619a9373d69b4cb003517686d057b5487101526efc8ec88: Status 404 returned error can't find the container with id 6f30c2cc6cb79dc28619a9373d69b4cb003517686d057b5487101526efc8ec88
Oct 30 22:19:14.210254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:14.210168    4581 generic.go:356] "Generic (PLEG): container finished" podID="dafbf2ee-8c6d-4897-a701-6f90ad2f3422" containerID="c6f8d87b4c2cb04684d44b9fb4d1a87ec2a2bf69d17a8aefcb7e75252442a096" exitCode=0
Oct 30 22:19:14.210369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:14.210324    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh" event={"ID":"dafbf2ee-8c6d-4897-a701-6f90ad2f3422","Type":"ContainerDied","Data":"c6f8d87b4c2cb04684d44b9fb4d1a87ec2a2bf69d17a8aefcb7e75252442a096"}
Oct 30 22:19:14.400290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:14.395780    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"]
Oct 30 22:19:14.400290 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:14.397365    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a2e504_9393_4da1_bd67_4b58fa4510a2.slice/crio-e4c6676d765126197e907bb879efe7457f5e47d2ad5e58e4ca36ee3d7dcb70ae WatchSource:0}: Error finding container e4c6676d765126197e907bb879efe7457f5e47d2ad5e58e4ca36ee3d7dcb70ae: Status 404 returned error can't find the container with id e4c6676d765126197e907bb879efe7457f5e47d2ad5e58e4ca36ee3d7dcb70ae
Oct 30 22:19:14.823991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:14.823959    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:19:14.824169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:14.824015    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 22:19:15.225027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.224973    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh" event={"ID":"a9a2e504-9393-4da1-bd67-4b58fa4510a2","Type":"ContainerStarted","Data":"925368464c7e84ea8b61a2884378efc2997f222072475cd4bff4ed9c3af09f26"}
Oct 30 22:19:15.225608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.225044    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh" event={"ID":"a9a2e504-9393-4da1-bd67-4b58fa4510a2","Type":"ContainerStarted","Data":"e4c6676d765126197e907bb879efe7457f5e47d2ad5e58e4ca36ee3d7dcb70ae"}
Oct 30 22:19:15.233587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.228862    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj" event={"ID":"d74c0295-9424-462a-95d9-7170777ed5e2","Type":"ContainerStarted","Data":"3e0f1db2b281bf9f05f0b07270ca381e4356c27d581f1cc28469afef18239bc9"}
Oct 30 22:19:15.233587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.228915    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj" event={"ID":"d74c0295-9424-462a-95d9-7170777ed5e2","Type":"ContainerStarted","Data":"6f30c2cc6cb79dc28619a9373d69b4cb003517686d057b5487101526efc8ec88"}
Oct 30 22:19:15.274246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.262468    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh" podStartSLOduration=2.262447225 podStartE2EDuration="2.262447225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:15.259406226 +0000 UTC m=+739.298345117" watchObservedRunningTime="2026-10-30 22:19:15.262447225 +0000 UTC m=+739.301386112"
Oct 30 22:19:15.306134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.306079    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj" podStartSLOduration=2.3060614409999998 podStartE2EDuration="2.306061441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:15.299642947 +0000 UTC m=+739.338581824" watchObservedRunningTime="2026-10-30 22:19:15.306061441 +0000 UTC m=+739.345000384"
Oct 30 22:19:15.451175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.451139    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:15.542721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.542629    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-xks-ca\") pod \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") "
Oct 30 22:19:15.542896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.542730    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-kube-api-access-rsdz5\" (UniqueName: \"kubernetes.io/projected/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-kube-api-access-rsdz5\") pod \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") "
Oct 30 22:19:15.542960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.542900    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-openshift-ca\") pod \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\" (UID: \"dafbf2ee-8c6d-4897-a701-6f90ad2f3422\") "
Oct 30 22:19:15.543278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.543252    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-openshift-ca" pod "dafbf2ee-8c6d-4897-a701-6f90ad2f3422" (UID: "dafbf2ee-8c6d-4897-a701-6f90ad2f3422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:15.543392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.543377    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:15.544949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.544924    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-xks-ca" pod "dafbf2ee-8c6d-4897-a701-6f90ad2f3422" (UID: "dafbf2ee-8c6d-4897-a701-6f90ad2f3422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:15.545337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.545312    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-kube-api-access-rsdz5" pod "dafbf2ee-8c6d-4897-a701-6f90ad2f3422" (UID: "dafbf2ee-8c6d-4897-a701-6f90ad2f3422"). InnerVolumeSpecName "kube-api-access-rsdz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:15.644506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.644467    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:15.644506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.644504    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsdz5\" (UniqueName: \"kubernetes.io/projected/dafbf2ee-8c6d-4897-a701-6f90ad2f3422-kube-api-access-rsdz5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:15.814339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.814260    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"]
Oct 30 22:19:15.814877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.814863    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dafbf2ee-8c6d-4897-a701-6f90ad2f3422" containerName="revoke-keys"
Oct 30 22:19:15.814925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.814880    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafbf2ee-8c6d-4897-a701-6f90ad2f3422" containerName="revoke-keys"
Oct 30 22:19:15.814996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.814986    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafbf2ee-8c6d-4897-a701-6f90ad2f3422" containerName="revoke-keys"
Oct 30 22:19:15.818546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.818531    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:15.828361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.828337    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:15.828640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.828618    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:15.829359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.829340    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:19:15.837506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.837481    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"]
Oct 30 22:19:15.948878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.948831    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gp5z\" (UniqueName: \"kubernetes.io/projected/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-kube-api-access-9gp5z\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:15.948878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.948881    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:15.949115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:15.949090    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:16.050136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.050105    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gp5z\" (UniqueName: \"kubernetes.io/projected/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-kube-api-access-9gp5z\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:16.050328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.050144    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:16.050328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.050268    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:16.050985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.050958    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:16.052748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.052724    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:16.067786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.067710    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gp5z\" (UniqueName: \"kubernetes.io/projected/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-kube-api-access-9gp5z\") pod \"maas-api-revoke-sub-0df6b8d3-45mqq\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:16.132560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.132531    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:16.245715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.245684    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"
Oct 30 22:19:16.246128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.245718    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh" event={"ID":"dafbf2ee-8c6d-4897-a701-6f90ad2f3422","Type":"ContainerDied","Data":"371b6a0cd884f994c755753bf8f01adc314237832e1c3e22043dcad4ffef579c"}
Oct 30 22:19:16.246128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.245769    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371b6a0cd884f994c755753bf8f01adc314237832e1c3e22043dcad4ffef579c"
Oct 30 22:19:16.247875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.247565    4581 generic.go:356] "Generic (PLEG): container finished" podID="a9a2e504-9393-4da1-bd67-4b58fa4510a2" containerID="925368464c7e84ea8b61a2884378efc2997f222072475cd4bff4ed9c3af09f26" exitCode=0
Oct 30 22:19:16.247875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.247685    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh" event={"ID":"a9a2e504-9393-4da1-bd67-4b58fa4510a2","Type":"ContainerDied","Data":"925368464c7e84ea8b61a2884378efc2997f222072475cd4bff4ed9c3af09f26"}
Oct 30 22:19:16.254732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.250451    4581 generic.go:356] "Generic (PLEG): container finished" podID="d74c0295-9424-462a-95d9-7170777ed5e2" containerID="3e0f1db2b281bf9f05f0b07270ca381e4356c27d581f1cc28469afef18239bc9" exitCode=0
Oct 30 22:19:16.254732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.250618    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj" event={"ID":"d74c0295-9424-462a-95d9-7170777ed5e2","Type":"ContainerDied","Data":"3e0f1db2b281bf9f05f0b07270ca381e4356c27d581f1cc28469afef18239bc9"}
Oct 30 22:19:16.451072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:16.451042    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"]
Oct 30 22:19:16.453527 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:16.453495    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c6a047_d46c_43cb_88b1_7b9f9b6576db.slice/crio-d33314890caf96ccc032532bb3a4220e9bc7dca77431f1fdd7bc98b9cecfb0e5 WatchSource:0}: Error finding container d33314890caf96ccc032532bb3a4220e9bc7dca77431f1fdd7bc98b9cecfb0e5: Status 404 returned error can't find the container with id d33314890caf96ccc032532bb3a4220e9bc7dca77431f1fdd7bc98b9cecfb0e5
Oct 30 22:19:17.262117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.262080    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq" event={"ID":"a7c6a047-d46c-43cb-88b1-7b9f9b6576db","Type":"ContainerStarted","Data":"9671168a4299ffca9a22726fa6f787ad542b1668bd522e848a14ae94d1e82975"}
Oct 30 22:19:17.262575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.262146    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq" event={"ID":"a7c6a047-d46c-43cb-88b1-7b9f9b6576db","Type":"ContainerStarted","Data":"d33314890caf96ccc032532bb3a4220e9bc7dca77431f1fdd7bc98b9cecfb0e5"}
Oct 30 22:19:17.302980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.302920    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq" podStartSLOduration=2.302904435 podStartE2EDuration="2.302904435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:17.298200709 +0000 UTC m=+741.337139594" watchObservedRunningTime="2026-10-30 22:19:17.302904435 +0000 UTC m=+741.341843337"
Oct 30 22:19:17.455723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.455688    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:17.511340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.511313    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:17.581770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.581665    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a2e504-9393-4da1-bd67-4b58fa4510a2-kube-api-access-lcc4v\" (UniqueName: \"kubernetes.io/projected/a9a2e504-9393-4da1-bd67-4b58fa4510a2-kube-api-access-lcc4v\") pod \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") "
Oct 30 22:19:17.581770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.581750    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-xks-ca\") pod \"d74c0295-9424-462a-95d9-7170777ed5e2\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") "
Oct 30 22:19:17.582022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.581831    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-openshift-ca\") pod \"d74c0295-9424-462a-95d9-7170777ed5e2\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") "
Oct 30 22:19:17.582022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.581865    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74c0295-9424-462a-95d9-7170777ed5e2-kube-api-access-24xjp\" (UniqueName: \"kubernetes.io/projected/d74c0295-9424-462a-95d9-7170777ed5e2-kube-api-access-24xjp\") pod \"d74c0295-9424-462a-95d9-7170777ed5e2\" (UID: \"d74c0295-9424-462a-95d9-7170777ed5e2\") "
Oct 30 22:19:17.582022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.581891    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-xks-ca\") pod \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") "
Oct 30 22:19:17.582022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.581943    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-openshift-ca\") pod \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\" (UID: \"a9a2e504-9393-4da1-bd67-4b58fa4510a2\") "
Oct 30 22:19:17.582446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.582416    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-openshift-ca" pod "d74c0295-9424-462a-95d9-7170777ed5e2" (UID: "d74c0295-9424-462a-95d9-7170777ed5e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:17.582969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.582668    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:17.583375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.583249    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-openshift-ca" pod "a9a2e504-9393-4da1-bd67-4b58fa4510a2" (UID: "a9a2e504-9393-4da1-bd67-4b58fa4510a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:17.585154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.585126    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a2e504-9393-4da1-bd67-4b58fa4510a2-kube-api-access-lcc4v" pod "a9a2e504-9393-4da1-bd67-4b58fa4510a2" (UID: "a9a2e504-9393-4da1-bd67-4b58fa4510a2"). InnerVolumeSpecName "kube-api-access-lcc4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:17.585856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.585826    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-xks-ca" pod "a9a2e504-9393-4da1-bd67-4b58fa4510a2" (UID: "a9a2e504-9393-4da1-bd67-4b58fa4510a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:17.585946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.585882    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-xks-ca" pod "d74c0295-9424-462a-95d9-7170777ed5e2" (UID: "d74c0295-9424-462a-95d9-7170777ed5e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:17.586043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.586025    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74c0295-9424-462a-95d9-7170777ed5e2-kube-api-access-24xjp" pod "d74c0295-9424-462a-95d9-7170777ed5e2" (UID: "d74c0295-9424-462a-95d9-7170777ed5e2"). InnerVolumeSpecName "kube-api-access-24xjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:17.684176 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.684143    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcc4v\" (UniqueName: \"kubernetes.io/projected/a9a2e504-9393-4da1-bd67-4b58fa4510a2-kube-api-access-lcc4v\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:17.684176 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.684173    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74c0295-9424-462a-95d9-7170777ed5e2-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:17.684176 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.684183    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24xjp\" (UniqueName: \"kubernetes.io/projected/d74c0295-9424-462a-95d9-7170777ed5e2-kube-api-access-24xjp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:17.684431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.684192    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:17.684431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:17.684200    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2e504-9393-4da1-bd67-4b58fa4510a2-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:18.273454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:18.273417    4581 generic.go:356] "Generic (PLEG): container finished" podID="a7c6a047-d46c-43cb-88b1-7b9f9b6576db" containerID="9671168a4299ffca9a22726fa6f787ad542b1668bd522e848a14ae94d1e82975" exitCode=0
Oct 30 22:19:18.273920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:18.273482    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq" event={"ID":"a7c6a047-d46c-43cb-88b1-7b9f9b6576db","Type":"ContainerDied","Data":"9671168a4299ffca9a22726fa6f787ad542b1668bd522e848a14ae94d1e82975"}
Oct 30 22:19:18.275288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:18.275258    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh" event={"ID":"a9a2e504-9393-4da1-bd67-4b58fa4510a2","Type":"ContainerDied","Data":"e4c6676d765126197e907bb879efe7457f5e47d2ad5e58e4ca36ee3d7dcb70ae"}
Oct 30 22:19:18.275423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:18.275315    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4c6676d765126197e907bb879efe7457f5e47d2ad5e58e4ca36ee3d7dcb70ae"
Oct 30 22:19:18.275423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:18.275273    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"
Oct 30 22:19:18.276989 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:18.276961    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj" event={"ID":"d74c0295-9424-462a-95d9-7170777ed5e2","Type":"ContainerDied","Data":"6f30c2cc6cb79dc28619a9373d69b4cb003517686d057b5487101526efc8ec88"}
Oct 30 22:19:18.277080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:18.277015    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f30c2cc6cb79dc28619a9373d69b4cb003517686d057b5487101526efc8ec88"
Oct 30 22:19:18.277080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:18.277074    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"
Oct 30 22:19:19.573696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.573665    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:19.639389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.636334    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-openshift-ca\") pod \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") "
Oct 30 22:19:19.639389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.636611    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-kube-api-access-9gp5z\" (UniqueName: \"kubernetes.io/projected/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-kube-api-access-9gp5z\") pod \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") "
Oct 30 22:19:19.639389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.636840    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-xks-ca\") pod \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\" (UID: \"a7c6a047-d46c-43cb-88b1-7b9f9b6576db\") "
Oct 30 22:19:19.677253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.675431    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-openshift-ca" pod "a7c6a047-d46c-43cb-88b1-7b9f9b6576db" (UID: "a7c6a047-d46c-43cb-88b1-7b9f9b6576db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:19.696449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.692403    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-xks-ca" pod "a7c6a047-d46c-43cb-88b1-7b9f9b6576db" (UID: "a7c6a047-d46c-43cb-88b1-7b9f9b6576db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:19.702354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.702317    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-kube-api-access-9gp5z" pod "a7c6a047-d46c-43cb-88b1-7b9f9b6576db" (UID: "a7c6a047-d46c-43cb-88b1-7b9f9b6576db"). InnerVolumeSpecName "kube-api-access-9gp5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:19.739276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.739234    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gp5z\" (UniqueName: \"kubernetes.io/projected/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-kube-api-access-9gp5z\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:19.739523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.739504    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:19.739625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:19.739612    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c6a047-d46c-43cb-88b1-7b9f9b6576db-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:20.342732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:20.342702    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"
Oct 30 22:19:20.342732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:20.342708    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq" event={"ID":"a7c6a047-d46c-43cb-88b1-7b9f9b6576db","Type":"ContainerDied","Data":"d33314890caf96ccc032532bb3a4220e9bc7dca77431f1fdd7bc98b9cecfb0e5"}
Oct 30 22:19:20.342931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:20.342769    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33314890caf96ccc032532bb3a4220e9bc7dca77431f1fdd7bc98b9cecfb0e5"
Oct 30 22:19:24.830438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.830398    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"]
Oct 30 22:19:24.831498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831465    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a7c6a047-d46c-43cb-88b1-7b9f9b6576db" containerName="revoke-keys"
Oct 30 22:19:24.831498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831491    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c6a047-d46c-43cb-88b1-7b9f9b6576db" containerName="revoke-keys"
Oct 30 22:19:24.831697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831529    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a9a2e504-9393-4da1-bd67-4b58fa4510a2" containerName="revoke-keys"
Oct 30 22:19:24.831697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831536    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a2e504-9393-4da1-bd67-4b58fa4510a2" containerName="revoke-keys"
Oct 30 22:19:24.831697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831547    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d74c0295-9424-462a-95d9-7170777ed5e2" containerName="revoke-keys"
Oct 30 22:19:24.831697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831553    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74c0295-9424-462a-95d9-7170777ed5e2" containerName="revoke-keys"
Oct 30 22:19:24.831697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831661    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74c0295-9424-462a-95d9-7170777ed5e2" containerName="revoke-keys"
Oct 30 22:19:24.831697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831681    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c6a047-d46c-43cb-88b1-7b9f9b6576db" containerName="revoke-keys"
Oct 30 22:19:24.831697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.831691    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a2e504-9393-4da1-bd67-4b58fa4510a2" containerName="revoke-keys"
Oct 30 22:19:24.836022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.835992    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:24.858025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.857994    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:24.861386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.861358    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:19:24.861566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.861374    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:24.865012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.864980    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"]
Oct 30 22:19:24.914621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.914585    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"]
Oct 30 22:19:24.915404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.915363    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:24.915587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.915560    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcqr\" (UniqueName: \"kubernetes.io/projected/3ce5b204-69a6-4e9e-b574-4aac213801ad-kube-api-access-vvcqr\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:24.915675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.915654    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:24.919671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.919648    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:24.943091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.942911    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"]
Oct 30 22:19:24.962563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:24.962534    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:25.019243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.016972    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.019243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.017123    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwk24\" (UniqueName: \"kubernetes.io/projected/b87e0e75-f6eb-46fa-893f-c59a07514953-kube-api-access-hwk24\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.019243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.017244    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.019243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.017334    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcqr\" (UniqueName: \"kubernetes.io/projected/3ce5b204-69a6-4e9e-b574-4aac213801ad-kube-api-access-vvcqr\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:25.019243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.017678    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:25.019243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.017777    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:25.019243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.019075    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:25.020905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.020853    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:25.037717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.037691    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcqr\" (UniqueName: \"kubernetes.io/projected/3ce5b204-69a6-4e9e-b574-4aac213801ad-kube-api-access-vvcqr\") pod \"maas-api-revoke-sub-05cea822-vqc78\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:25.119372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.119335    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.119565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.119387    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwk24\" (UniqueName: \"kubernetes.io/projected/b87e0e75-f6eb-46fa-893f-c59a07514953-kube-api-access-hwk24\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.119565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.119430    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.120234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.120187    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.122463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.122431    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.137064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.137037    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwk24\" (UniqueName: \"kubernetes.io/projected/b87e0e75-f6eb-46fa-893f-c59a07514953-kube-api-access-hwk24\") pod \"maas-api-revoke-sub-8439b6d2-rv2k4\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.154879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.154853    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:25.238553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.238360    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:25.489862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.489785    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"]
Oct 30 22:19:25.495141 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:25.495111    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce5b204_69a6_4e9e_b574_4aac213801ad.slice/crio-256d309be62d384675c14dd042102507e1f6d2bca35be95e14ae33e81d3d1435 WatchSource:0}: Error finding container 256d309be62d384675c14dd042102507e1f6d2bca35be95e14ae33e81d3d1435: Status 404 returned error can't find the container with id 256d309be62d384675c14dd042102507e1f6d2bca35be95e14ae33e81d3d1435
Oct 30 22:19:25.589901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:25.589862    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"]
Oct 30 22:19:26.142114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.137337    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"]
Oct 30 22:19:26.142808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.142785    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.163803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.163776    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"]
Oct 30 22:19:26.247270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.246086    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.247270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.246142    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cww4\" (UniqueName: \"kubernetes.io/projected/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-kube-api-access-5cww4\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.247270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.246235    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.347238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.347175    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.347518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.347254    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cww4\" (UniqueName: \"kubernetes.io/projected/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-kube-api-access-5cww4\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.347518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.347315    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.348046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.348020    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.354232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.354190    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.368074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.368046    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cww4\" (UniqueName: \"kubernetes.io/projected/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-kube-api-access-5cww4\") pod \"maas-api-revoke-sub-d5ba606e-9php9\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.399992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.399861    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78" event={"ID":"3ce5b204-69a6-4e9e-b574-4aac213801ad","Type":"ContainerStarted","Data":"427828c0f7434cc762367c0077183ba5449823a68b23d6e2265eb6c057656064"}
Oct 30 22:19:26.399992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.399930    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78" event={"ID":"3ce5b204-69a6-4e9e-b574-4aac213801ad","Type":"ContainerStarted","Data":"256d309be62d384675c14dd042102507e1f6d2bca35be95e14ae33e81d3d1435"}
Oct 30 22:19:26.401775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.401750    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4" event={"ID":"b87e0e75-f6eb-46fa-893f-c59a07514953","Type":"ContainerStarted","Data":"83c1d20857a2b4f1aa4beddda0a5ae840ca7236573f3ff6d24b2f809ddc75e88"}
Oct 30 22:19:26.401894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.401789    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4" event={"ID":"b87e0e75-f6eb-46fa-893f-c59a07514953","Type":"ContainerStarted","Data":"04d3a0c53dde7e75ec2ea894057d7bde32a76d0226f2dbac415068b70a4187b3"}
Oct 30 22:19:26.444239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.443933    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78" podStartSLOduration=2.443916369 podStartE2EDuration="2.443916369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:26.437015017 +0000 UTC m=+750.475953907" watchObservedRunningTime="2026-10-30 22:19:26.443916369 +0000 UTC m=+750.482855256"
Oct 30 22:19:26.461616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.461587    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:26.475594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:26.475521    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4" podStartSLOduration=2.4755020070000002 podStartE2EDuration="2.475502007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:26.47296957 +0000 UTC m=+750.511908464" watchObservedRunningTime="2026-10-30 22:19:26.475502007 +0000 UTC m=+750.514440893"
Oct 30 22:19:27.411025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:27.410992    4581 generic.go:356] "Generic (PLEG): container finished" podID="3ce5b204-69a6-4e9e-b574-4aac213801ad" containerID="427828c0f7434cc762367c0077183ba5449823a68b23d6e2265eb6c057656064" exitCode=0
Oct 30 22:19:27.411509 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:27.411076    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78" event={"ID":"3ce5b204-69a6-4e9e-b574-4aac213801ad","Type":"ContainerDied","Data":"427828c0f7434cc762367c0077183ba5449823a68b23d6e2265eb6c057656064"}
Oct 30 22:19:27.412851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:27.412824    4581 generic.go:356] "Generic (PLEG): container finished" podID="b87e0e75-f6eb-46fa-893f-c59a07514953" containerID="83c1d20857a2b4f1aa4beddda0a5ae840ca7236573f3ff6d24b2f809ddc75e88" exitCode=0
Oct 30 22:19:27.412963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:27.412907    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4" event={"ID":"b87e0e75-f6eb-46fa-893f-c59a07514953","Type":"ContainerDied","Data":"83c1d20857a2b4f1aa4beddda0a5ae840ca7236573f3ff6d24b2f809ddc75e88"}
Oct 30 22:19:27.654727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:27.654694    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"]
Oct 30 22:19:27.669607 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:27.669568    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5ac55e_fbc2_4a53_b3ac_1911b9e28ae6.slice/crio-150cefd68711b57902258b036ea3f89fcf25d081907460846c34769a55e26ee1 WatchSource:0}: Error finding container 150cefd68711b57902258b036ea3f89fcf25d081907460846c34769a55e26ee1: Status 404 returned error can't find the container with id 150cefd68711b57902258b036ea3f89fcf25d081907460846c34769a55e26ee1
Oct 30 22:19:28.422300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.422259    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9" event={"ID":"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6","Type":"ContainerStarted","Data":"f475c44e5c69a0f00d61aa9fc4259627b5b45375128388f00cd452d429ca56e4"}
Oct 30 22:19:28.422723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.422328    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9" event={"ID":"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6","Type":"ContainerStarted","Data":"150cefd68711b57902258b036ea3f89fcf25d081907460846c34769a55e26ee1"}
Oct 30 22:19:28.463042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.462503    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9" podStartSLOduration=2.462484178 podStartE2EDuration="2.462484178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:28.456136552 +0000 UTC m=+752.495075440" watchObservedRunningTime="2026-10-30 22:19:28.462484178 +0000 UTC m=+752.501423065"
Oct 30 22:19:28.681499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.681432    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:28.685060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.685030    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:28.779723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.779682    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-xks-ca\") pod \"3ce5b204-69a6-4e9e-b574-4aac213801ad\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") "
Oct 30 22:19:28.779895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.779741    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-openshift-ca\") pod \"3ce5b204-69a6-4e9e-b574-4aac213801ad\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") "
Oct 30 22:19:28.779895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.779829    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-openshift-ca\") pod \"b87e0e75-f6eb-46fa-893f-c59a07514953\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") "
Oct 30 22:19:28.780027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.779933    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-xks-ca\") pod \"b87e0e75-f6eb-46fa-893f-c59a07514953\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") "
Oct 30 22:19:28.780027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.779963    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce5b204-69a6-4e9e-b574-4aac213801ad-kube-api-access-vvcqr\" (UniqueName: \"kubernetes.io/projected/3ce5b204-69a6-4e9e-b574-4aac213801ad-kube-api-access-vvcqr\") pod \"3ce5b204-69a6-4e9e-b574-4aac213801ad\" (UID: \"3ce5b204-69a6-4e9e-b574-4aac213801ad\") "
Oct 30 22:19:28.780125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.780024    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87e0e75-f6eb-46fa-893f-c59a07514953-kube-api-access-hwk24\" (UniqueName: \"kubernetes.io/projected/b87e0e75-f6eb-46fa-893f-c59a07514953-kube-api-access-hwk24\") pod \"b87e0e75-f6eb-46fa-893f-c59a07514953\" (UID: \"b87e0e75-f6eb-46fa-893f-c59a07514953\") "
Oct 30 22:19:28.780268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.780246    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-openshift-ca" pod "b87e0e75-f6eb-46fa-893f-c59a07514953" (UID: "b87e0e75-f6eb-46fa-893f-c59a07514953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:28.780489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.780474    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:28.781088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.781032    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-openshift-ca" pod "3ce5b204-69a6-4e9e-b574-4aac213801ad" (UID: "3ce5b204-69a6-4e9e-b574-4aac213801ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:28.782798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.782759    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-xks-ca" pod "3ce5b204-69a6-4e9e-b574-4aac213801ad" (UID: "3ce5b204-69a6-4e9e-b574-4aac213801ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:28.783291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.783268    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87e0e75-f6eb-46fa-893f-c59a07514953-kube-api-access-hwk24" pod "b87e0e75-f6eb-46fa-893f-c59a07514953" (UID: "b87e0e75-f6eb-46fa-893f-c59a07514953"). InnerVolumeSpecName "kube-api-access-hwk24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:28.783532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.783508    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-xks-ca" pod "b87e0e75-f6eb-46fa-893f-c59a07514953" (UID: "b87e0e75-f6eb-46fa-893f-c59a07514953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:28.783622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.783599    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce5b204-69a6-4e9e-b574-4aac213801ad-kube-api-access-vvcqr" pod "3ce5b204-69a6-4e9e-b574-4aac213801ad" (UID: "3ce5b204-69a6-4e9e-b574-4aac213801ad"). InnerVolumeSpecName "kube-api-access-vvcqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:28.881459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.881427    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87e0e75-f6eb-46fa-893f-c59a07514953-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:28.881459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.881458    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvcqr\" (UniqueName: \"kubernetes.io/projected/3ce5b204-69a6-4e9e-b574-4aac213801ad-kube-api-access-vvcqr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:28.881660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.881475    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwk24\" (UniqueName: \"kubernetes.io/projected/b87e0e75-f6eb-46fa-893f-c59a07514953-kube-api-access-hwk24\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:28.881660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.881488    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:28.881660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:28.881504    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce5b204-69a6-4e9e-b574-4aac213801ad-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:29.186241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.185385    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"]
Oct 30 22:19:29.189241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.187632    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3ce5b204-69a6-4e9e-b574-4aac213801ad" containerName="revoke-keys"
Oct 30 22:19:29.189241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.187656    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce5b204-69a6-4e9e-b574-4aac213801ad" containerName="revoke-keys"
Oct 30 22:19:29.189241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.187686    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b87e0e75-f6eb-46fa-893f-c59a07514953" containerName="revoke-keys"
Oct 30 22:19:29.189241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.187695    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87e0e75-f6eb-46fa-893f-c59a07514953" containerName="revoke-keys"
Oct 30 22:19:29.189241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.187832    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce5b204-69a6-4e9e-b574-4aac213801ad" containerName="revoke-keys"
Oct 30 22:19:29.189241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.187849    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87e0e75-f6eb-46fa-893f-c59a07514953" containerName="revoke-keys"
Oct 30 22:19:29.204642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.204584    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.217054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.212634    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"]
Oct 30 22:19:29.287680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.287645    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95n6\" (UniqueName: \"kubernetes.io/projected/9c38381f-e41e-44de-bbb0-19da746b43ca-kube-api-access-v95n6\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.287991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.287967    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.288179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.288162    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.389982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.389943    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95n6\" (UniqueName: \"kubernetes.io/projected/9c38381f-e41e-44de-bbb0-19da746b43ca-kube-api-access-v95n6\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.390191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.390018    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.390191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.390056    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.390703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.390681    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.392588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.392564    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.408427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.408404    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95n6\" (UniqueName: \"kubernetes.io/projected/9c38381f-e41e-44de-bbb0-19da746b43ca-kube-api-access-v95n6\") pod \"maas-api-revoke-sub-589cf440-md8w7\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.431473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.431447    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"
Oct 30 22:19:29.431911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.431444    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78" event={"ID":"3ce5b204-69a6-4e9e-b574-4aac213801ad","Type":"ContainerDied","Data":"256d309be62d384675c14dd042102507e1f6d2bca35be95e14ae33e81d3d1435"}
Oct 30 22:19:29.431911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.431576    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="256d309be62d384675c14dd042102507e1f6d2bca35be95e14ae33e81d3d1435"
Oct 30 22:19:29.439550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.439482    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"
Oct 30 22:19:29.439720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.439699    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4" event={"ID":"b87e0e75-f6eb-46fa-893f-c59a07514953","Type":"ContainerDied","Data":"04d3a0c53dde7e75ec2ea894057d7bde32a76d0226f2dbac415068b70a4187b3"}
Oct 30 22:19:29.439838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.439824    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d3a0c53dde7e75ec2ea894057d7bde32a76d0226f2dbac415068b70a4187b3"
Oct 30 22:19:29.441892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.441684    4581 generic.go:356] "Generic (PLEG): container finished" podID="5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6" containerID="f475c44e5c69a0f00d61aa9fc4259627b5b45375128388f00cd452d429ca56e4" exitCode=0
Oct 30 22:19:29.441892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.441712    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9" event={"ID":"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6","Type":"ContainerDied","Data":"f475c44e5c69a0f00d61aa9fc4259627b5b45375128388f00cd452d429ca56e4"}
Oct 30 22:19:29.525978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.525950    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"]
Oct 30 22:19:29.530257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.530240    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.541531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.541503    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:29.549997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.549971    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"]
Oct 30 22:19:29.694135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.694047    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.694303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.694211    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.694406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.694384    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7x64\" (UniqueName: \"kubernetes.io/projected/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-kube-api-access-x7x64\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.796137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.796101    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7x64\" (UniqueName: \"kubernetes.io/projected/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-kube-api-access-x7x64\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.796358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.796190    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.796358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.796271    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.797031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.796962    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.798821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.798798    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.813462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.813431    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7x64\" (UniqueName: \"kubernetes.io/projected/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-kube-api-access-x7x64\") pod \"maas-api-revoke-sub-b8eac990-mbjjd\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.844067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.844035    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:29.858405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:29.858379    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"]
Oct 30 22:19:29.861124 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:29.861083    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c38381f_e41e_44de_bbb0_19da746b43ca.slice/crio-556306cafdaca4bcffb7b8aebb7ed1b318f46f8ff739fc65b1b4511d91b858d1 WatchSource:0}: Error finding container 556306cafdaca4bcffb7b8aebb7ed1b318f46f8ff739fc65b1b4511d91b858d1: Status 404 returned error can't find the container with id 556306cafdaca4bcffb7b8aebb7ed1b318f46f8ff739fc65b1b4511d91b858d1
Oct 30 22:19:30.163657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.163624    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"]
Oct 30 22:19:30.165704 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:30.165660    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81ebe40_59bb_4bb4_9d9a_c94698a04efa.slice/crio-25fa117940875de603824a3cf761ee05ba2127fd0bf09a66cd2b8f4d1f54c16f WatchSource:0}: Error finding container 25fa117940875de603824a3cf761ee05ba2127fd0bf09a66cd2b8f4d1f54c16f: Status 404 returned error can't find the container with id 25fa117940875de603824a3cf761ee05ba2127fd0bf09a66cd2b8f4d1f54c16f
Oct 30 22:19:30.451574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.451537    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7" event={"ID":"9c38381f-e41e-44de-bbb0-19da746b43ca","Type":"ContainerStarted","Data":"41a57df7a8cbc97a1916fe90ad437b04aaf79f93032e1488bdf366a6653ee32e"}
Oct 30 22:19:30.452088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.451604    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7" event={"ID":"9c38381f-e41e-44de-bbb0-19da746b43ca","Type":"ContainerStarted","Data":"556306cafdaca4bcffb7b8aebb7ed1b318f46f8ff739fc65b1b4511d91b858d1"}
Oct 30 22:19:30.453382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.453344    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd" event={"ID":"b81ebe40-59bb-4bb4-9d9a-c94698a04efa","Type":"ContainerStarted","Data":"25fa117940875de603824a3cf761ee05ba2127fd0bf09a66cd2b8f4d1f54c16f"}
Oct 30 22:19:30.485794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.485731    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7" podStartSLOduration=1.485711788 podStartE2EDuration="1.485711788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:30.48457204 +0000 UTC m=+754.523510927" watchObservedRunningTime="2026-10-30 22:19:30.485711788 +0000 UTC m=+754.524650677"
Oct 30 22:19:30.673839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.673807    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:30.809890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.809850    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-kube-api-access-5cww4\" (UniqueName: \"kubernetes.io/projected/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-kube-api-access-5cww4\") pod \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") "
Oct 30 22:19:30.810110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.810088    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-openshift-ca\") pod \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") "
Oct 30 22:19:30.810290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.810117    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-xks-ca\") pod \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\" (UID: \"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6\") "
Oct 30 22:19:30.810537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.810510    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-openshift-ca" pod "5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6" (UID: "5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:30.810616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.810590    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:30.812297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.812267    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-kube-api-access-5cww4" pod "5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6" (UID: "5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6"). InnerVolumeSpecName "kube-api-access-5cww4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:30.812504 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.812488    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-xks-ca" pod "5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6" (UID: "5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:30.830524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.830496    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"]
Oct 30 22:19:30.831079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.831061    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6" containerName="revoke-keys"
Oct 30 22:19:30.831079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.831080    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6" containerName="revoke-keys"
Oct 30 22:19:30.831246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.831204    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6" containerName="revoke-keys"
Oct 30 22:19:30.834858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.834832    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:30.856627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.856601    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"]
Oct 30 22:19:30.911957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.911884    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cww4\" (UniqueName: \"kubernetes.io/projected/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-kube-api-access-5cww4\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:30.911957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:30.911916    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:31.013190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.013152    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.013410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.013250    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.013410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.013382    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9m4\" (UniqueName: \"kubernetes.io/projected/d100333d-7032-45a7-abf3-108111c7ced1-kube-api-access-6f9m4\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.114063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.114030    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.114253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.114085    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9m4\" (UniqueName: \"kubernetes.io/projected/d100333d-7032-45a7-abf3-108111c7ced1-kube-api-access-6f9m4\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.114343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.114288    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.114739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.114718    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.116957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.116936    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.131840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.131815    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9m4\" (UniqueName: \"kubernetes.io/projected/d100333d-7032-45a7-abf3-108111c7ced1-kube-api-access-6f9m4\") pod \"maas-api-revoke-sub-63ac8f84-cdzjh\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.148789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.148764    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:31.456661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.456632    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"]
Oct 30 22:19:31.457912 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:31.457882    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd100333d_7032_45a7_abf3_108111c7ced1.slice/crio-39c0b74088e78564ed90cd7bc76374078978a48a4b4abd463a036b65c15e5f55 WatchSource:0}: Error finding container 39c0b74088e78564ed90cd7bc76374078978a48a4b4abd463a036b65c15e5f55: Status 404 returned error can't find the container with id 39c0b74088e78564ed90cd7bc76374078978a48a4b4abd463a036b65c15e5f55
Oct 30 22:19:31.465963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.465064    4581 generic.go:356] "Generic (PLEG): container finished" podID="9c38381f-e41e-44de-bbb0-19da746b43ca" containerID="41a57df7a8cbc97a1916fe90ad437b04aaf79f93032e1488bdf366a6653ee32e" exitCode=0
Oct 30 22:19:31.465963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.465256    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7" event={"ID":"9c38381f-e41e-44de-bbb0-19da746b43ca","Type":"ContainerDied","Data":"41a57df7a8cbc97a1916fe90ad437b04aaf79f93032e1488bdf366a6653ee32e"}
Oct 30 22:19:31.467752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.467731    4581 generic.go:356] "Generic (PLEG): container finished" podID="b81ebe40-59bb-4bb4-9d9a-c94698a04efa" containerID="df31263be56d90577547bfbc16dffbbd48fc8e0cce7920f496c8d7332ec71c94" exitCode=0
Oct 30 22:19:31.467861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.467808    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd" event={"ID":"b81ebe40-59bb-4bb4-9d9a-c94698a04efa","Type":"ContainerDied","Data":"df31263be56d90577547bfbc16dffbbd48fc8e0cce7920f496c8d7332ec71c94"}
Oct 30 22:19:31.469459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.469441    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9" event={"ID":"5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6","Type":"ContainerDied","Data":"150cefd68711b57902258b036ea3f89fcf25d081907460846c34769a55e26ee1"}
Oct 30 22:19:31.469540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.469480    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"
Oct 30 22:19:31.469596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:31.469482    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="150cefd68711b57902258b036ea3f89fcf25d081907460846c34769a55e26ee1"
Oct 30 22:19:32.478657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.478626    4581 generic.go:356] "Generic (PLEG): container finished" podID="d100333d-7032-45a7-abf3-108111c7ced1" containerID="5529460fae754847dd23316eba7a1fe13c740f41160c5a20510949b383a3a4ee" exitCode=0
Oct 30 22:19:32.479137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.478710    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh" event={"ID":"d100333d-7032-45a7-abf3-108111c7ced1","Type":"ContainerDied","Data":"5529460fae754847dd23316eba7a1fe13c740f41160c5a20510949b383a3a4ee"}
Oct 30 22:19:32.479137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.478766    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh" event={"ID":"d100333d-7032-45a7-abf3-108111c7ced1","Type":"ContainerStarted","Data":"39c0b74088e78564ed90cd7bc76374078978a48a4b4abd463a036b65c15e5f55"}
Oct 30 22:19:32.675420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.675391    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:32.702384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.702352    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:32.740689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.740660    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c38381f-e41e-44de-bbb0-19da746b43ca-kube-api-access-v95n6\" (UniqueName: \"kubernetes.io/projected/9c38381f-e41e-44de-bbb0-19da746b43ca-kube-api-access-v95n6\") pod \"9c38381f-e41e-44de-bbb0-19da746b43ca\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") "
Oct 30 22:19:32.740854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.740730    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-kube-api-access-x7x64\" (UniqueName: \"kubernetes.io/projected/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-kube-api-access-x7x64\") pod \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") "
Oct 30 22:19:32.740854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.740779    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-xks-ca\") pod \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") "
Oct 30 22:19:32.743173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.743137    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-xks-ca" pod "b81ebe40-59bb-4bb4-9d9a-c94698a04efa" (UID: "b81ebe40-59bb-4bb4-9d9a-c94698a04efa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:32.743173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.743155    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-kube-api-access-x7x64" pod "b81ebe40-59bb-4bb4-9d9a-c94698a04efa" (UID: "b81ebe40-59bb-4bb4-9d9a-c94698a04efa"). InnerVolumeSpecName "kube-api-access-x7x64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:32.743453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.743433    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c38381f-e41e-44de-bbb0-19da746b43ca-kube-api-access-v95n6" pod "9c38381f-e41e-44de-bbb0-19da746b43ca" (UID: "9c38381f-e41e-44de-bbb0-19da746b43ca"). InnerVolumeSpecName "kube-api-access-v95n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:32.842333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842245    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-openshift-ca\") pod \"9c38381f-e41e-44de-bbb0-19da746b43ca\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") "
Oct 30 22:19:32.842333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842309    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-openshift-ca\") pod \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\" (UID: \"b81ebe40-59bb-4bb4-9d9a-c94698a04efa\") "
Oct 30 22:19:32.842572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842429    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-xks-ca\") pod \"9c38381f-e41e-44de-bbb0-19da746b43ca\" (UID: \"9c38381f-e41e-44de-bbb0-19da746b43ca\") "
Oct 30 22:19:32.842677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842645    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-openshift-ca" pod "9c38381f-e41e-44de-bbb0-19da746b43ca" (UID: "9c38381f-e41e-44de-bbb0-19da746b43ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:32.842726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842652    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-openshift-ca" pod "b81ebe40-59bb-4bb4-9d9a-c94698a04efa" (UID: "b81ebe40-59bb-4bb4-9d9a-c94698a04efa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:32.842859 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842844    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:32.842898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842863    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:32.842898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842872    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:32.842898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842881    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v95n6\" (UniqueName: \"kubernetes.io/projected/9c38381f-e41e-44de-bbb0-19da746b43ca-kube-api-access-v95n6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:32.842898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.842890    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7x64\" (UniqueName: \"kubernetes.io/projected/b81ebe40-59bb-4bb4-9d9a-c94698a04efa-kube-api-access-x7x64\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:32.844627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.844606    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-xks-ca" pod "9c38381f-e41e-44de-bbb0-19da746b43ca" (UID: "9c38381f-e41e-44de-bbb0-19da746b43ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:32.944372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:32.944340    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c38381f-e41e-44de-bbb0-19da746b43ca-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:33.491519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.491488    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"
Oct 30 22:19:33.491998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.491519    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7" event={"ID":"9c38381f-e41e-44de-bbb0-19da746b43ca","Type":"ContainerDied","Data":"556306cafdaca4bcffb7b8aebb7ed1b318f46f8ff739fc65b1b4511d91b858d1"}
Oct 30 22:19:33.491998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.491611    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="556306cafdaca4bcffb7b8aebb7ed1b318f46f8ff739fc65b1b4511d91b858d1"
Oct 30 22:19:33.493671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.493650    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"
Oct 30 22:19:33.493894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.493668    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd" event={"ID":"b81ebe40-59bb-4bb4-9d9a-c94698a04efa","Type":"ContainerDied","Data":"25fa117940875de603824a3cf761ee05ba2127fd0bf09a66cd2b8f4d1f54c16f"}
Oct 30 22:19:33.494044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.493942    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25fa117940875de603824a3cf761ee05ba2127fd0bf09a66cd2b8f4d1f54c16f"
Oct 30 22:19:33.664643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.664609    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:33.753095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.752998    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d100333d-7032-45a7-abf3-108111c7ced1-kube-api-access-6f9m4\" (UniqueName: \"kubernetes.io/projected/d100333d-7032-45a7-abf3-108111c7ced1-kube-api-access-6f9m4\") pod \"d100333d-7032-45a7-abf3-108111c7ced1\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") "
Oct 30 22:19:33.753281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.753182    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-xks-ca\") pod \"d100333d-7032-45a7-abf3-108111c7ced1\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") "
Oct 30 22:19:33.753405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.753301    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-openshift-ca\") pod \"d100333d-7032-45a7-abf3-108111c7ced1\" (UID: \"d100333d-7032-45a7-abf3-108111c7ced1\") "
Oct 30 22:19:33.753682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.753658    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-openshift-ca" pod "d100333d-7032-45a7-abf3-108111c7ced1" (UID: "d100333d-7032-45a7-abf3-108111c7ced1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:33.755951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.755923    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-xks-ca" pod "d100333d-7032-45a7-abf3-108111c7ced1" (UID: "d100333d-7032-45a7-abf3-108111c7ced1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:33.756359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.756334    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d100333d-7032-45a7-abf3-108111c7ced1-kube-api-access-6f9m4" pod "d100333d-7032-45a7-abf3-108111c7ced1" (UID: "d100333d-7032-45a7-abf3-108111c7ced1"). InnerVolumeSpecName "kube-api-access-6f9m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:33.855073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.855033    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:33.855073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.855074    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f9m4\" (UniqueName: \"kubernetes.io/projected/d100333d-7032-45a7-abf3-108111c7ced1-kube-api-access-6f9m4\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:33.855318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:33.855088    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d100333d-7032-45a7-abf3-108111c7ced1-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:34.503862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:34.503830    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh" event={"ID":"d100333d-7032-45a7-abf3-108111c7ced1","Type":"ContainerDied","Data":"39c0b74088e78564ed90cd7bc76374078978a48a4b4abd463a036b65c15e5f55"}
Oct 30 22:19:34.503862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:34.503863    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"
Oct 30 22:19:34.504339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:34.503879    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c0b74088e78564ed90cd7bc76374078978a48a4b4abd463a036b65c15e5f55"
Oct 30 22:19:36.491886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.491676    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"]
Oct 30 22:19:36.499273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.499246    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b81ebe40-59bb-4bb4-9d9a-c94698a04efa" containerName="revoke-keys"
Oct 30 22:19:36.499515 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.499496    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81ebe40-59bb-4bb4-9d9a-c94698a04efa" containerName="revoke-keys"
Oct 30 22:19:36.499631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.499619    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d100333d-7032-45a7-abf3-108111c7ced1" containerName="revoke-keys"
Oct 30 22:19:36.499717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.499706    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d100333d-7032-45a7-abf3-108111c7ced1" containerName="revoke-keys"
Oct 30 22:19:36.499870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.499838    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9c38381f-e41e-44de-bbb0-19da746b43ca" containerName="revoke-keys"
Oct 30 22:19:36.499959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.499948    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c38381f-e41e-44de-bbb0-19da746b43ca" containerName="revoke-keys"
Oct 30 22:19:36.500200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.500185    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c38381f-e41e-44de-bbb0-19da746b43ca" containerName="revoke-keys"
Oct 30 22:19:36.503940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.503921    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81ebe40-59bb-4bb4-9d9a-c94698a04efa" containerName="revoke-keys"
Oct 30 22:19:36.504914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.504894    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d100333d-7032-45a7-abf3-108111c7ced1" containerName="revoke-keys"
Oct 30 22:19:36.694245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.687428    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"]
Oct 30 22:19:36.694245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.687566    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.784307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.765612    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdzd\" (UniqueName: \"kubernetes.io/projected/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-kube-api-access-sxdzd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.977242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.959728    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.991178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.991141    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.991518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.991472    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.991704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.991686    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.991940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.991923    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.992083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.992067    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.992239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.992203    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:36.992388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:36.992373    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.096069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.095965    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.096369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.096345    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.096526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.096510    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.096685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.096662    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.096853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.096820    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdzd\" (UniqueName: \"kubernetes.io/projected/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-kube-api-access-sxdzd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.097036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.097006    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.097178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.097163    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.097371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.097340    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.098047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.098026    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.098647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.098627    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.100195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.100174    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.116010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.115984    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.116327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.116194    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.127670 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.127631    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.135252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.134359    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.145250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.142413    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.156876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.156810    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.166974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.166906    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdzd\" (UniqueName: \"kubernetes.io/projected/9c8d4001-3885-49c0-84b2-1bcb032aa3d5-kube-api-access-sxdzd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-24vbj\" (UID: \"9c8d4001-3885-49c0-84b2-1bcb032aa3d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.220245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.219600    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:37.618879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.618852    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"]
Oct 30 22:19:37.619788 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:37.619761    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8d4001_3885_49c0_84b2_1bcb032aa3d5.slice/crio-a1c7a2f0fc496ed94ad055a6c11c1cf63e70c73c4fa9c746dddadb41f135ac7a WatchSource:0}: Error finding container a1c7a2f0fc496ed94ad055a6c11c1cf63e70c73c4fa9c746dddadb41f135ac7a: Status 404 returned error can't find the container with id a1c7a2f0fc496ed94ad055a6c11c1cf63e70c73c4fa9c746dddadb41f135ac7a
Oct 30 22:19:37.622576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.622536    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:37.622668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.622620    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:37.622725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:37.622664    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:38.619239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:38.619182    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" event={"ID":"9c8d4001-3885-49c0-84b2-1bcb032aa3d5","Type":"ContainerStarted","Data":"040f88260042a7fa35865cfec8ee482d6ee54bf6f945d896578f12dd51291e44"}
Oct 30 22:19:38.619784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:38.619270    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" event={"ID":"9c8d4001-3885-49c0-84b2-1bcb032aa3d5","Type":"ContainerStarted","Data":"a1c7a2f0fc496ed94ad055a6c11c1cf63e70c73c4fa9c746dddadb41f135ac7a"}
Oct 30 22:19:38.690243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:38.689423    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podStartSLOduration=2.689403026 podStartE2EDuration="2.689403026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:38.669756971 +0000 UTC m=+762.708695858" watchObservedRunningTime="2026-10-30 22:19:38.689403026 +0000 UTC m=+762.728341913"
Oct 30 22:19:39.221074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.221045    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:39.222532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.222506    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:19:39.222647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.222555    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:19:39.849038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.848997    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"]
Oct 30 22:19:39.871547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.871515    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:39.892624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.891064    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"]
Oct 30 22:19:39.892624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.891758    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:39.902991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.901730    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:39.902991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.901823    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:19:39.902991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.901863    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:39.969963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.969918    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lgc\" (UniqueName: \"kubernetes.io/projected/7acf4990-0751-41f6-ad33-ac085c2f6891-kube-api-access-95lgc\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:39.970391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.970365    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:39.970599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:39.970583    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:40.077244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.072967    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:40.077244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.073087    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:40.077244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.073162    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lgc\" (UniqueName: \"kubernetes.io/projected/7acf4990-0751-41f6-ad33-ac085c2f6891-kube-api-access-95lgc\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:40.077244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.074131    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:40.100858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.100775    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:40.104992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.104963    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lgc\" (UniqueName: \"kubernetes.io/projected/7acf4990-0751-41f6-ad33-ac085c2f6891-kube-api-access-95lgc\") pod \"maas-api-revoke-sub-67206247-k5ts4\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:40.194125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.191271    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"]
Oct 30 22:19:40.196458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.196428    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"]
Oct 30 22:19:40.196605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.196585    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.207426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.207020    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:40.221610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.221577    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:19:40.221707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.221640    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:19:40.284279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.281970    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.284279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.282113    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.284279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.282150    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtcm\" (UniqueName: \"kubernetes.io/projected/6f284056-f73a-462b-ac37-4c245abd1486-kube-api-access-svtcm\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.394327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.389238    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.394327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.389374    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.394327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.389413    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtcm\" (UniqueName: \"kubernetes.io/projected/6f284056-f73a-462b-ac37-4c245abd1486-kube-api-access-svtcm\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.394327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.391365    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.408077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.408010    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtcm\" (UniqueName: \"kubernetes.io/projected/6f284056-f73a-462b-ac37-4c245abd1486-kube-api-access-svtcm\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.410678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.410649    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec8d756-tswds\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.517192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.517158    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:40.602349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.602299    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"]
Oct 30 22:19:40.638453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.638425    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4" event={"ID":"7acf4990-0751-41f6-ad33-ac085c2f6891","Type":"ContainerStarted","Data":"74a3d725cd70a10ddee051cffd9123efa62ac0f7d9212d6f2dc30b35d61ec629"}
Oct 30 22:19:40.820469 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:19:40.820433    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:19:40.892582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:40.892417    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"]
Oct 30 22:19:40.895626 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:40.895598    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f284056_f73a_462b_ac37_4c245abd1486.slice/crio-ec5a163fa9270ef4cbb4e724c588b4d31d5c1b314923c4c04ce6da5e82bea5f7 WatchSource:0}: Error finding container ec5a163fa9270ef4cbb4e724c588b4d31d5c1b314923c4c04ce6da5e82bea5f7: Status 404 returned error can't find the container with id ec5a163fa9270ef4cbb4e724c588b4d31d5c1b314923c4c04ce6da5e82bea5f7
Oct 30 22:19:41.222232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.221977    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:19:41.222232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.222028    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:19:41.649778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.649741    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4" event={"ID":"7acf4990-0751-41f6-ad33-ac085c2f6891","Type":"ContainerStarted","Data":"1472ba5389f0173f1b49986ba1cf72400cae06edef154e336c984f4db6363fff"}
Oct 30 22:19:41.675285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.672737    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds" event={"ID":"6f284056-f73a-462b-ac37-4c245abd1486","Type":"ContainerStarted","Data":"731a6b3ec2829fc79c6e85723372b73fee3dd24897b709532614ff1b9ff9f695"}
Oct 30 22:19:41.675285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.672804    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds" event={"ID":"6f284056-f73a-462b-ac37-4c245abd1486","Type":"ContainerStarted","Data":"ec5a163fa9270ef4cbb4e724c588b4d31d5c1b314923c4c04ce6da5e82bea5f7"}
Oct 30 22:19:41.694005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.692984    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4" podStartSLOduration=2.692969819 podStartE2EDuration="2.692969819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:41.690361753 +0000 UTC m=+765.729300648" watchObservedRunningTime="2026-10-30 22:19:41.692969819 +0000 UTC m=+765.731908704"
Oct 30 22:19:41.725521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.725468    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds" podStartSLOduration=1.725452211 podStartE2EDuration="1.725452211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:41.722650097 +0000 UTC m=+765.761588983" watchObservedRunningTime="2026-10-30 22:19:41.725452211 +0000 UTC m=+765.764391097"
Oct 30 22:19:41.820642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.820610    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"]
Oct 30 22:19:41.824806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.824786    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:41.842936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.842783    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"]
Oct 30 22:19:41.912955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.912864    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdv8b\" (UniqueName: \"kubernetes.io/projected/01c467ff-a8f7-4274-b395-c7323ec211ec-kube-api-access-pdv8b\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:41.913347 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.913165    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:41.913399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:41.913348    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:42.014442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.014390    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:42.014612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.014493    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdv8b\" (UniqueName: \"kubernetes.io/projected/01c467ff-a8f7-4274-b395-c7323ec211ec-kube-api-access-pdv8b\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:42.014653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.014642    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:42.015362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.015342    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:42.017073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.017049    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:42.031744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.031709    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdv8b\" (UniqueName: \"kubernetes.io/projected/01c467ff-a8f7-4274-b395-c7323ec211ec-kube-api-access-pdv8b\") pod \"maas-api-revoke-sub-872ebf2a-btg2w\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:42.139174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.139143    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:42.221647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.221614    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:19:42.221815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.221678    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:19:42.481077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.481034    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"]
Oct 30 22:19:42.688686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.688647    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w" event={"ID":"01c467ff-a8f7-4274-b395-c7323ec211ec","Type":"ContainerStarted","Data":"f7b68597abd236eaf64741a860b10f22c7ac795d04049b1f312101fedf23435b"}
Oct 30 22:19:42.690755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.690728    4581 generic.go:356] "Generic (PLEG): container finished" podID="7acf4990-0751-41f6-ad33-ac085c2f6891" containerID="1472ba5389f0173f1b49986ba1cf72400cae06edef154e336c984f4db6363fff" exitCode=0
Oct 30 22:19:42.690866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.690801    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4" event={"ID":"7acf4990-0751-41f6-ad33-ac085c2f6891","Type":"ContainerDied","Data":"1472ba5389f0173f1b49986ba1cf72400cae06edef154e336c984f4db6363fff"}
Oct 30 22:19:42.692490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.692466    4581 generic.go:356] "Generic (PLEG): container finished" podID="6f284056-f73a-462b-ac37-4c245abd1486" containerID="731a6b3ec2829fc79c6e85723372b73fee3dd24897b709532614ff1b9ff9f695" exitCode=0
Oct 30 22:19:42.692686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:42.692665    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds" event={"ID":"6f284056-f73a-462b-ac37-4c245abd1486","Type":"ContainerDied","Data":"731a6b3ec2829fc79c6e85723372b73fee3dd24897b709532614ff1b9ff9f695"}
Oct 30 22:19:43.220837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:43.220737    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:19:43.220837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:43.220799    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:19:43.706076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:43.706039    4581 generic.go:356] "Generic (PLEG): container finished" podID="01c467ff-a8f7-4274-b395-c7323ec211ec" containerID="b58e5aca517c1b5db48b1b39b8d68193e9713d26afbcc17327d0d6f0c6a35b59" exitCode=0
Oct 30 22:19:43.706303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:43.706096    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w" event={"ID":"01c467ff-a8f7-4274-b395-c7323ec211ec","Type":"ContainerDied","Data":"b58e5aca517c1b5db48b1b39b8d68193e9713d26afbcc17327d0d6f0c6a35b59"}
Oct 30 22:19:43.942970 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:43.942653    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:44.032454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.032416    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:44.042289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.042075    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-xks-ca\") pod \"6f284056-f73a-462b-ac37-4c245abd1486\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") "
Oct 30 22:19:44.042289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.042126    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-openshift-ca\") pod \"6f284056-f73a-462b-ac37-4c245abd1486\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") "
Oct 30 22:19:44.042289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.042212    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f284056-f73a-462b-ac37-4c245abd1486-kube-api-access-svtcm\" (UniqueName: \"kubernetes.io/projected/6f284056-f73a-462b-ac37-4c245abd1486-kube-api-access-svtcm\") pod \"6f284056-f73a-462b-ac37-4c245abd1486\" (UID: \"6f284056-f73a-462b-ac37-4c245abd1486\") "
Oct 30 22:19:44.042986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.042954    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-openshift-ca" pod "6f284056-f73a-462b-ac37-4c245abd1486" (UID: "6f284056-f73a-462b-ac37-4c245abd1486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:44.045453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.045427    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-xks-ca" pod "6f284056-f73a-462b-ac37-4c245abd1486" (UID: "6f284056-f73a-462b-ac37-4c245abd1486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:44.045599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.045572    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f284056-f73a-462b-ac37-4c245abd1486-kube-api-access-svtcm" pod "6f284056-f73a-462b-ac37-4c245abd1486" (UID: "6f284056-f73a-462b-ac37-4c245abd1486"). InnerVolumeSpecName "kube-api-access-svtcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:44.143514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.143482    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-xks-ca\") pod \"7acf4990-0751-41f6-ad33-ac085c2f6891\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") "
Oct 30 22:19:44.143701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.143569    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-openshift-ca\") pod \"7acf4990-0751-41f6-ad33-ac085c2f6891\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") "
Oct 30 22:19:44.143701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.143659    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acf4990-0751-41f6-ad33-ac085c2f6891-kube-api-access-95lgc\" (UniqueName: \"kubernetes.io/projected/7acf4990-0751-41f6-ad33-ac085c2f6891-kube-api-access-95lgc\") pod \"7acf4990-0751-41f6-ad33-ac085c2f6891\" (UID: \"7acf4990-0751-41f6-ad33-ac085c2f6891\") "
Oct 30 22:19:44.144256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.144188    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtcm\" (UniqueName: \"kubernetes.io/projected/6f284056-f73a-462b-ac37-4c245abd1486-kube-api-access-svtcm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:44.144256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.144234    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:44.144256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.144250    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f284056-f73a-462b-ac37-4c245abd1486-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:44.144503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.144381    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-openshift-ca" pod "7acf4990-0751-41f6-ad33-ac085c2f6891" (UID: "7acf4990-0751-41f6-ad33-ac085c2f6891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:44.147119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.147086    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-xks-ca" pod "7acf4990-0751-41f6-ad33-ac085c2f6891" (UID: "7acf4990-0751-41f6-ad33-ac085c2f6891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:44.147246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.147140    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acf4990-0751-41f6-ad33-ac085c2f6891-kube-api-access-95lgc" pod "7acf4990-0751-41f6-ad33-ac085c2f6891" (UID: "7acf4990-0751-41f6-ad33-ac085c2f6891"). InnerVolumeSpecName "kube-api-access-95lgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:44.222283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.220954    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:19:44.222283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.221026    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:19:44.245893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.245862    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:44.245893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.245890    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acf4990-0751-41f6-ad33-ac085c2f6891-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:44.245893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.245899    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lgc\" (UniqueName: \"kubernetes.io/projected/7acf4990-0751-41f6-ad33-ac085c2f6891-kube-api-access-95lgc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:44.715901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.715860    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4" event={"ID":"7acf4990-0751-41f6-ad33-ac085c2f6891","Type":"ContainerDied","Data":"74a3d725cd70a10ddee051cffd9123efa62ac0f7d9212d6f2dc30b35d61ec629"}
Oct 30 22:19:44.716099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.715930    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a3d725cd70a10ddee051cffd9123efa62ac0f7d9212d6f2dc30b35d61ec629"
Oct 30 22:19:44.716099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.715878    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"
Oct 30 22:19:44.717467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.717433    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds" event={"ID":"6f284056-f73a-462b-ac37-4c245abd1486","Type":"ContainerDied","Data":"ec5a163fa9270ef4cbb4e724c588b4d31d5c1b314923c4c04ce6da5e82bea5f7"}
Oct 30 22:19:44.717623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.717493    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5a163fa9270ef4cbb4e724c588b4d31d5c1b314923c4c04ce6da5e82bea5f7"
Oct 30 22:19:44.717623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.717456    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"
Oct 30 22:19:44.896777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.896753    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:44.955164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.955122    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-xks-ca\") pod \"01c467ff-a8f7-4274-b395-c7323ec211ec\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") "
Oct 30 22:19:44.955421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.955395    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c467ff-a8f7-4274-b395-c7323ec211ec-kube-api-access-pdv8b\" (UniqueName: \"kubernetes.io/projected/01c467ff-a8f7-4274-b395-c7323ec211ec-kube-api-access-pdv8b\") pod \"01c467ff-a8f7-4274-b395-c7323ec211ec\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") "
Oct 30 22:19:44.955867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.955815    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-openshift-ca\") pod \"01c467ff-a8f7-4274-b395-c7323ec211ec\" (UID: \"01c467ff-a8f7-4274-b395-c7323ec211ec\") "
Oct 30 22:19:44.956381 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.956243    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-openshift-ca" pod "01c467ff-a8f7-4274-b395-c7323ec211ec" (UID: "01c467ff-a8f7-4274-b395-c7323ec211ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:44.956886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.956860    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:44.970426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.966117    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-xks-ca" pod "01c467ff-a8f7-4274-b395-c7323ec211ec" (UID: "01c467ff-a8f7-4274-b395-c7323ec211ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:44.970426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:44.966148    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c467ff-a8f7-4274-b395-c7323ec211ec-kube-api-access-pdv8b" pod "01c467ff-a8f7-4274-b395-c7323ec211ec" (UID: "01c467ff-a8f7-4274-b395-c7323ec211ec"). InnerVolumeSpecName "kube-api-access-pdv8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:45.058515 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:45.058466    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdv8b\" (UniqueName: \"kubernetes.io/projected/01c467ff-a8f7-4274-b395-c7323ec211ec-kube-api-access-pdv8b\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:45.058515 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:45.058513    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c467ff-a8f7-4274-b395-c7323ec211ec-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:45.221107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:45.221015    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:19:45.221107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:45.221070    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 22:19:45.751092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:45.751051    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"
Oct 30 22:19:45.751539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:45.751058    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w" event={"ID":"01c467ff-a8f7-4274-b395-c7323ec211ec","Type":"ContainerDied","Data":"f7b68597abd236eaf64741a860b10f22c7ac795d04049b1f312101fedf23435b"}
Oct 30 22:19:45.751539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:45.751210    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7b68597abd236eaf64741a860b10f22c7ac795d04049b1f312101fedf23435b"
Oct 30 22:19:46.790083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:46.790046    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:46.791412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:46.791383    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:46.793369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:46.793346    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj"
Oct 30 22:19:50.592916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.592878    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"]
Oct 30 22:19:50.593476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593461    4581 cpu_manager.go:414] "containerMap: removing container" podUID="01c467ff-a8f7-4274-b395-c7323ec211ec" containerName="revoke-keys"
Oct 30 22:19:50.593522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593477    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c467ff-a8f7-4274-b395-c7323ec211ec" containerName="revoke-keys"
Oct 30 22:19:50.593522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593496    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6f284056-f73a-462b-ac37-4c245abd1486" containerName="revoke-keys"
Oct 30 22:19:50.593522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593501    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f284056-f73a-462b-ac37-4c245abd1486" containerName="revoke-keys"
Oct 30 22:19:50.593522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593514    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7acf4990-0751-41f6-ad33-ac085c2f6891" containerName="revoke-keys"
Oct 30 22:19:50.593522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593519    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acf4990-0751-41f6-ad33-ac085c2f6891" containerName="revoke-keys"
Oct 30 22:19:50.593671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593591    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c467ff-a8f7-4274-b395-c7323ec211ec" containerName="revoke-keys"
Oct 30 22:19:50.593671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593600    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f284056-f73a-462b-ac37-4c245abd1486" containerName="revoke-keys"
Oct 30 22:19:50.593671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.593607    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acf4990-0751-41f6-ad33-ac085c2f6891" containerName="revoke-keys"
Oct 30 22:19:50.598620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.598594    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.612409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.612379    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:50.621134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.621085    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"]
Oct 30 22:19:50.621588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.621540    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:50.621588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.621576    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:50.621753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.621578    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:19:50.725636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.725597    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.725636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.725641    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnj4\" (UniqueName: \"kubernetes.io/projected/2ea62c3d-ccb5-4999-b152-04010e5487b9-kube-api-access-zcnj4\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.725922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.725786    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.827523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.827486    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.827723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.827698    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.827797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.827737    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnj4\" (UniqueName: \"kubernetes.io/projected/2ea62c3d-ccb5-4999-b152-04010e5487b9-kube-api-access-zcnj4\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.828442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.828409    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.830141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.830118    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.846784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.846709    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnj4\" (UniqueName: \"kubernetes.io/projected/2ea62c3d-ccb5-4999-b152-04010e5487b9-kube-api-access-zcnj4\") pod \"maas-api-revoke-sub-e3d2d135-p6pfb\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:50.920474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:50.920432    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:51.364239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.358015    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"]
Oct 30 22:19:51.397579 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:51.394078    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea62c3d_ccb5_4999_b152_04010e5487b9.slice/crio-fbe9a0de50197ea7e80d3d414ff5381252a42e1684bbe788a32b38be14c4141c WatchSource:0}: Error finding container fbe9a0de50197ea7e80d3d414ff5381252a42e1684bbe788a32b38be14c4141c: Status 404 returned error can't find the container with id fbe9a0de50197ea7e80d3d414ff5381252a42e1684bbe788a32b38be14c4141c
Oct 30 22:19:51.520992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.516748    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"]
Oct 30 22:19:51.526531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.526499    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.634521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.634488    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"]
Oct 30 22:19:51.677000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.676956    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.692044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.691991    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3b5e99-a183-44e8-a4ea-34113aac767b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.692954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.692423    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.693187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.693157    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.693368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.693348    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.693508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.693493    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.693627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.693610    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4sq\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-kube-api-access-7q4sq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.693769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.693753    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.694267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.693962    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799201    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3b5e99-a183-44e8-a4ea-34113aac767b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799344    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799388    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799423    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799462    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799493    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4sq\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-kube-api-access-7q4sq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799546    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799643    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.800241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.799713    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.824244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.806775    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.824244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.807499    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3b5e99-a183-44e8-a4ea-34113aac767b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.824244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.808116    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.824244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.808414    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.824244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.809008    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.824244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.817874    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.824244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.823880    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.868246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.856081    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb" event={"ID":"2ea62c3d-ccb5-4999-b152-04010e5487b9","Type":"ContainerStarted","Data":"fbe9a0de50197ea7e80d3d414ff5381252a42e1684bbe788a32b38be14c4141c"}
Oct 30 22:19:51.887315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.887164    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4sq\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-kube-api-access-7q4sq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.897311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.890531    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vbv8b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:51.950284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:51.948459    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:52.628088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.628059    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"]
Oct 30 22:19:52.630390 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:52.630352    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3b5e99_a183_44e8_a4ea_34113aac767b.slice/crio-d220aa80f809e27f03056857a44f5c0903e054a60b3ee9d15e9a46bc6c7f57f3 WatchSource:0}: Error finding container d220aa80f809e27f03056857a44f5c0903e054a60b3ee9d15e9a46bc6c7f57f3: Status 404 returned error can't find the container with id d220aa80f809e27f03056857a44f5c0903e054a60b3ee9d15e9a46bc6c7f57f3
Oct 30 22:19:52.632888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.632856    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:52.632967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.632926    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:52.633005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.632964    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:19:52.871005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.870974    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb" event={"ID":"2ea62c3d-ccb5-4999-b152-04010e5487b9","Type":"ContainerStarted","Data":"c56e27c8a6affdbba67e1cba124d3446921d088ce32b505905657f654d267a15"}
Oct 30 22:19:52.873437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.873408    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" event={"ID":"1f3b5e99-a183-44e8-a4ea-34113aac767b","Type":"ContainerStarted","Data":"ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390"}
Oct 30 22:19:52.873567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.873457    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" event={"ID":"1f3b5e99-a183-44e8-a4ea-34113aac767b","Type":"ContainerStarted","Data":"d220aa80f809e27f03056857a44f5c0903e054a60b3ee9d15e9a46bc6c7f57f3"}
Oct 30 22:19:52.949924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.949807    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:19:52.951203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:52.951147    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podStartSLOduration=1.951133258 podStartE2EDuration="1.951133258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:52.947320394 +0000 UTC m=+776.986259280" watchObservedRunningTime="2026-10-30 22:19:52.951133258 +0000 UTC m=+776.990072143"
Oct 30 22:19:53.884284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:53.884250    4581 generic.go:356] "Generic (PLEG): container finished" podID="2ea62c3d-ccb5-4999-b152-04010e5487b9" containerID="c56e27c8a6affdbba67e1cba124d3446921d088ce32b505905657f654d267a15" exitCode=0
Oct 30 22:19:53.884692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:53.884343    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb" event={"ID":"2ea62c3d-ccb5-4999-b152-04010e5487b9","Type":"ContainerDied","Data":"c56e27c8a6affdbba67e1cba124d3446921d088ce32b505905657f654d267a15"}
Oct 30 22:19:53.978370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:53.968371    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vbv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:19:53.978370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:53.968438    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:19:54.110088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.110058    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:54.138702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.138620    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-xks-ca\") pod \"2ea62c3d-ccb5-4999-b152-04010e5487b9\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") "
Oct 30 22:19:54.138856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.138723    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-openshift-ca\") pod \"2ea62c3d-ccb5-4999-b152-04010e5487b9\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") "
Oct 30 22:19:54.138856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.138757    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea62c3d-ccb5-4999-b152-04010e5487b9-kube-api-access-zcnj4\" (UniqueName: \"kubernetes.io/projected/2ea62c3d-ccb5-4999-b152-04010e5487b9-kube-api-access-zcnj4\") pod \"2ea62c3d-ccb5-4999-b152-04010e5487b9\" (UID: \"2ea62c3d-ccb5-4999-b152-04010e5487b9\") "
Oct 30 22:19:54.139134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.139109    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-openshift-ca" pod "2ea62c3d-ccb5-4999-b152-04010e5487b9" (UID: "2ea62c3d-ccb5-4999-b152-04010e5487b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:54.140858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.140833    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-xks-ca" pod "2ea62c3d-ccb5-4999-b152-04010e5487b9" (UID: "2ea62c3d-ccb5-4999-b152-04010e5487b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:54.140969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.140949    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea62c3d-ccb5-4999-b152-04010e5487b9-kube-api-access-zcnj4" pod "2ea62c3d-ccb5-4999-b152-04010e5487b9" (UID: "2ea62c3d-ccb5-4999-b152-04010e5487b9"). InnerVolumeSpecName "kube-api-access-zcnj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:54.239689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.239664    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:54.239689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.239686    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea62c3d-ccb5-4999-b152-04010e5487b9-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:54.239689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.239697    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcnj4\" (UniqueName: \"kubernetes.io/projected/2ea62c3d-ccb5-4999-b152-04010e5487b9-kube-api-access-zcnj4\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:19:54.895450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.895415    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb" event={"ID":"2ea62c3d-ccb5-4999-b152-04010e5487b9","Type":"ContainerDied","Data":"fbe9a0de50197ea7e80d3d414ff5381252a42e1684bbe788a32b38be14c4141c"}
Oct 30 22:19:54.895450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.895439    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"
Oct 30 22:19:54.896009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.895471    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbe9a0de50197ea7e80d3d414ff5381252a42e1684bbe788a32b38be14c4141c"
Oct 30 22:19:54.949786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.949749    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vbv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:19:54.949938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:54.949833    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:19:55.949535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:55.949501    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vbv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:19:55.949943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:55.949563    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:19:56.212314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.212236    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"]
Oct 30 22:19:56.212807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.212794    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2ea62c3d-ccb5-4999-b152-04010e5487b9" containerName="revoke-keys"
Oct 30 22:19:56.212862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.212809    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea62c3d-ccb5-4999-b152-04010e5487b9" containerName="revoke-keys"
Oct 30 22:19:56.212899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.212893    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea62c3d-ccb5-4999-b152-04010e5487b9" containerName="revoke-keys"
Oct 30 22:19:56.216834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.216814    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.241982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.241955    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:56.242135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.242022    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:56.242135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.242031    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:56.242135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.242084    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:19:56.248690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.248658    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"]
Oct 30 22:19:56.279824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.279779    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llhv\" (UniqueName: \"kubernetes.io/projected/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-kube-api-access-7llhv\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.280026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.279842    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.280026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.279945    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.381196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.381154    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.381425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.381326    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llhv\" (UniqueName: \"kubernetes.io/projected/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-kube-api-access-7llhv\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.381425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.381359    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.381910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.381852    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.384009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.383980    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.399611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.399589    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llhv\" (UniqueName: \"kubernetes.io/projected/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-kube-api-access-7llhv\") pod \"maas-api-revoke-sub-81bc550b-wccbz\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.531368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.531283    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:19:56.879479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.879437    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"]
Oct 30 22:19:56.880593 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:56.880568    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e71675_907e_4fcd_b602_d6bdaeb1fdb6.slice/crio-44649b142528b5a09833285393ec184adbb8e7eb187e4eb42dc691618a3385a8 WatchSource:0}: Error finding container 44649b142528b5a09833285393ec184adbb8e7eb187e4eb42dc691618a3385a8: Status 404 returned error can't find the container with id 44649b142528b5a09833285393ec184adbb8e7eb187e4eb42dc691618a3385a8
Oct 30 22:19:56.913178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.913146    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz" event={"ID":"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6","Type":"ContainerStarted","Data":"44649b142528b5a09833285393ec184adbb8e7eb187e4eb42dc691618a3385a8"}
Oct 30 22:19:56.949740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.949706    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vbv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:19:56.950184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:56.949764    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:19:57.922676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:57.922635    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz" event={"ID":"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6","Type":"ContainerStarted","Data":"7bc2e614c5d1d4bc878879545eacb614b5a9fa26b230ab7826b171a12805adff"}
Oct 30 22:19:57.949416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:57.949384    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vbv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:19:57.949612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:57.949454    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:19:57.969095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:57.969022    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz" podStartSLOduration=1.96900118 podStartE2EDuration="1.96900118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:57.95567037 +0000 UTC m=+781.994609257" watchObservedRunningTime="2026-10-30 22:19:57.96900118 +0000 UTC m=+782.007940067"
Oct 30 22:19:58.033477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.033443    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"]
Oct 30 22:19:58.037547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.037526    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.083472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.083370    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"]
Oct 30 22:19:58.102110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.102069    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.102370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.102118    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvnc\" (UniqueName: \"kubernetes.io/projected/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-kube-api-access-4mvnc\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.102370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.102322    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.203177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.203086    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.203177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.203132    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvnc\" (UniqueName: \"kubernetes.io/projected/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-kube-api-access-4mvnc\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.203428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.203259    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.203905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.203885    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.205835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.205816    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.220744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.220710    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvnc\" (UniqueName: \"kubernetes.io/projected/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-kube-api-access-4mvnc\") pod \"maas-api-revoke-sub-27a1841b-bxz7k\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.375275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.368725    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:19:58.697517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.697490    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"]
Oct 30 22:19:58.709460 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:19:58.709422    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bb5f43_ecac_4845_b8c7_69abfac4d07c.slice/crio-45b65ead91f7ef9ac8e77b5f934a8fb7b6ee5a84dcfa362313a41bb532038720 WatchSource:0}: Error finding container 45b65ead91f7ef9ac8e77b5f934a8fb7b6ee5a84dcfa362313a41bb532038720: Status 404 returned error can't find the container with id 45b65ead91f7ef9ac8e77b5f934a8fb7b6ee5a84dcfa362313a41bb532038720
Oct 30 22:19:58.932718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.932678    4581 generic.go:356] "Generic (PLEG): container finished" podID="a6e71675-907e-4fcd-b602-d6bdaeb1fdb6" containerID="7bc2e614c5d1d4bc878879545eacb614b5a9fa26b230ab7826b171a12805adff" exitCode=0
Oct 30 22:19:58.933117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.932761    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz" event={"ID":"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6","Type":"ContainerDied","Data":"7bc2e614c5d1d4bc878879545eacb614b5a9fa26b230ab7826b171a12805adff"}
Oct 30 22:19:58.934324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.934292    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k" event={"ID":"b6bb5f43-ecac-4845-b8c7-69abfac4d07c","Type":"ContainerStarted","Data":"45b65ead91f7ef9ac8e77b5f934a8fb7b6ee5a84dcfa362313a41bb532038720"}
Oct 30 22:19:58.949879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.949800    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vbv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:19:58.949879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:58.949855    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:19:59.946611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:59.946523    4581 generic.go:356] "Generic (PLEG): container finished" podID="b6bb5f43-ecac-4845-b8c7-69abfac4d07c" containerID="a18ca81caea4b67cc3c284c721fae7a997ad029d9f7458154dd166a5ebea685a" exitCode=0
Oct 30 22:19:59.946993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:59.946599    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k" event={"ID":"b6bb5f43-ecac-4845-b8c7-69abfac4d07c","Type":"ContainerDied","Data":"a18ca81caea4b67cc3c284c721fae7a997ad029d9f7458154dd166a5ebea685a"}
Oct 30 22:19:59.949403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:59.949369    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vbv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 22:19:59.949521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:19:59.949428    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 22:20:00.149028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.148996    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:20:00.236244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.236119    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-openshift-ca\") pod \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") "
Oct 30 22:20:00.236244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.236182    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-kube-api-access-7llhv\" (UniqueName: \"kubernetes.io/projected/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-kube-api-access-7llhv\") pod \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") "
Oct 30 22:20:00.236244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.236236    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-xks-ca\") pod \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\" (UID: \"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6\") "
Oct 30 22:20:00.236550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.236506    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-openshift-ca" pod "a6e71675-907e-4fcd-b602-d6bdaeb1fdb6" (UID: "a6e71675-907e-4fcd-b602-d6bdaeb1fdb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:00.236657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.236641    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:20:00.238598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.238568    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-xks-ca" pod "a6e71675-907e-4fcd-b602-d6bdaeb1fdb6" (UID: "a6e71675-907e-4fcd-b602-d6bdaeb1fdb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:00.238598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.238577    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-kube-api-access-7llhv" pod "a6e71675-907e-4fcd-b602-d6bdaeb1fdb6" (UID: "a6e71675-907e-4fcd-b602-d6bdaeb1fdb6"). InnerVolumeSpecName "kube-api-access-7llhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:00.337669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.337635    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llhv\" (UniqueName: \"kubernetes.io/projected/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-kube-api-access-7llhv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:20:00.337669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.337668    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:20:00.643963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.643820    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"]
Oct 30 22:20:00.645982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.645950    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a6e71675-907e-4fcd-b602-d6bdaeb1fdb6" containerName="revoke-keys"
Oct 30 22:20:00.645982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.645981    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e71675-907e-4fcd-b602-d6bdaeb1fdb6" containerName="revoke-keys"
Oct 30 22:20:00.646397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.646174    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e71675-907e-4fcd-b602-d6bdaeb1fdb6" containerName="revoke-keys"
Oct 30 22:20:00.650831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.650788    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.676657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.676625    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"distinct-w6-012ba0-kserve-self-signed-certs\""
Oct 30 22:20:00.684249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.684199    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"]
Oct 30 22:20:00.749571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.749527    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-tmp-dir\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.749780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.749624    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf0e91-e65c-4c67-bd21-5274367f24af-tls-certs\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.749780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.749669    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-dshm\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.749780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.749703    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-home\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.749963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.749841    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-model-cache\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.750019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.749959    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsflv\" (UniqueName: \"kubernetes.io/projected/bbbf0e91-e65c-4c67-bd21-5274367f24af-kube-api-access-lsflv\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.851544    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-model-cache\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.851659    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsflv\" (UniqueName: \"kubernetes.io/projected/bbbf0e91-e65c-4c67-bd21-5274367f24af-kube-api-access-lsflv\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.851702    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-tmp-dir\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.851736    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf0e91-e65c-4c67-bd21-5274367f24af-tls-certs\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.851767    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-dshm\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.851801    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-home\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.852248    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-home\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.852600    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-model-cache\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.853559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.853137    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-tmp-dir\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.887139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.882089    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsflv\" (UniqueName: \"kubernetes.io/projected/bbbf0e91-e65c-4c67-bd21-5274367f24af-kube-api-access-lsflv\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.887139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.882257    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-dshm\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.908087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.897094    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf0e91-e65c-4c67-bd21-5274367f24af-tls-certs\") pod \"distinct-w6-012ba0-kserve-85d876d77c-h977d\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:00.958450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.958410    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz" event={"ID":"a6e71675-907e-4fcd-b602-d6bdaeb1fdb6","Type":"ContainerDied","Data":"44649b142528b5a09833285393ec184adbb8e7eb187e4eb42dc691618a3385a8"}
Oct 30 22:20:00.958924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.958476    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44649b142528b5a09833285393ec184adbb8e7eb187e4eb42dc691618a3385a8"
Oct 30 22:20:00.958924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.958436    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"
Oct 30 22:20:00.982877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:00.982836    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:01.160102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.159993    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:20:01.272049 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.272003    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-openshift-ca\") pod \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") "
Oct 30 22:20:01.272258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.272096    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-kube-api-access-4mvnc\" (UniqueName: \"kubernetes.io/projected/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-kube-api-access-4mvnc\") pod \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") "
Oct 30 22:20:01.272258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.272147    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-xks-ca\") pod \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\" (UID: \"b6bb5f43-ecac-4845-b8c7-69abfac4d07c\") "
Oct 30 22:20:01.272559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.272527    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-openshift-ca" pod "b6bb5f43-ecac-4845-b8c7-69abfac4d07c" (UID: "b6bb5f43-ecac-4845-b8c7-69abfac4d07c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:01.272752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.272732    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:20:01.274852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.274814    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-kube-api-access-4mvnc" pod "b6bb5f43-ecac-4845-b8c7-69abfac4d07c" (UID: "b6bb5f43-ecac-4845-b8c7-69abfac4d07c"). InnerVolumeSpecName "kube-api-access-4mvnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:01.275286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.275258    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-xks-ca" pod "b6bb5f43-ecac-4845-b8c7-69abfac4d07c" (UID: "b6bb5f43-ecac-4845-b8c7-69abfac4d07c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:01.348312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.347263    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"]
Oct 30 22:20:01.374000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.373955    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mvnc\" (UniqueName: \"kubernetes.io/projected/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-kube-api-access-4mvnc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:20:01.374000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.374002    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6bb5f43-ecac-4845-b8c7-69abfac4d07c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:20:01.530459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.530430    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:20:01.530926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.530863    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:20:01.531784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.531766    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:20:01.969093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.969060    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"
Oct 30 22:20:01.969093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.969084    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k" event={"ID":"b6bb5f43-ecac-4845-b8c7-69abfac4d07c","Type":"ContainerDied","Data":"45b65ead91f7ef9ac8e77b5f934a8fb7b6ee5a84dcfa362313a41bb532038720"}
Oct 30 22:20:01.969631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.969143    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45b65ead91f7ef9ac8e77b5f934a8fb7b6ee5a84dcfa362313a41bb532038720"
Oct 30 22:20:01.975141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.974617    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" event={"ID":"bbbf0e91-e65c-4c67-bd21-5274367f24af","Type":"ContainerStarted","Data":"5a87c4bbabb5b2f3df50a208c453c3c981a5e6c494fc4d75ff01fe3967ac4925"}
Oct 30 22:20:01.975141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.974672    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" event={"ID":"bbbf0e91-e65c-4c67-bd21-5274367f24af","Type":"ContainerStarted","Data":"e2228442c605a3649c016f190fe9e1c53bbb97c3646597b75b98acf0622e231b"}
Oct 30 22:20:01.983708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:01.983685    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:02.012011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:02.011942    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podStartSLOduration=2.011922354 podStartE2EDuration="2.011922354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:02.009885931 +0000 UTC m=+786.048824818" watchObservedRunningTime="2026-10-30 22:20:02.011922354 +0000 UTC m=+786.050861241"
Oct 30 22:20:07.519286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.519243    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"]
Oct 30 22:20:07.519794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.519781    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b6bb5f43-ecac-4845-b8c7-69abfac4d07c" containerName="revoke-keys"
Oct 30 22:20:07.519853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.519795    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bb5f43-ecac-4845-b8c7-69abfac4d07c" containerName="revoke-keys"
Oct 30 22:20:07.519929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.519921    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bb5f43-ecac-4845-b8c7-69abfac4d07c" containerName="revoke-keys"
Oct 30 22:20:07.523733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.523716    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.555638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.555606    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"]
Oct 30 22:20:07.666798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.666761    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldp9q\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-kube-api-access-ldp9q\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.666995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.666819    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.666995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.666914    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.666995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.666936    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.666995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.666954    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.667160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.667070    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.667160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.667106    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.667271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.667235    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.667271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.667262    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.768809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.768765    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldp9q\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-kube-api-access-ldp9q\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.768994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.768854    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.768994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.768884    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.768994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.768912    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769032    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769099    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769130    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769252    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769281    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769426    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769501    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769589    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769680    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.769866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.769846    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.771782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.771757    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.771925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.771906    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.787250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.787176    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.787375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.787362    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldp9q\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-kube-api-access-ldp9q\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.838172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.838133    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:07.980450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.980403    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"]
Oct 30 22:20:07.986741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.986702    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:07.999319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:07.999287    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-2b1e88-openshift-default-dockercfg-ljrwv\""
Oct 30 22:20:08.004945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.004915    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"]
Oct 30 22:20:08.073892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.073793    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.074036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.073927    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.074036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.073963    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntzn\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-kube-api-access-qntzn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.074140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.074053    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.074286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.074171    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.074286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.074232    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.074369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.074302    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.074406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.074369    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.074442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.074410    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.175672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.175638    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.175806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.175710    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.175947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.175925    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.176147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.176130    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.176364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.176347    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.176497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.176476    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntzn\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-kube-api-access-qntzn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.176610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.176584    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.176730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.176712    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.176894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.176877    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.177205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.177179    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.177506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.177483    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.177718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.177577    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"]
Oct 30 22:20:08.177815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.177792    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.177941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.177914    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.178440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.178416    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.179636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.179609    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.180585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.180520    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.181483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.181449    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:08.181575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.181530    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:08.181639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.181576    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:08.195982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.195954    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.196179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.196098    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntzn\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-kube-api-access-qntzn\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.302227 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.302178    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:08.635749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.635719    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"]
Oct 30 22:20:08.637210 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:20:08.637182    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887a1cf8_96c9_444f_a7cd_6847a0c28f74.slice/crio-cf4822e38064fa3ea2c40beae3522f947aa954c0dd3fe4b234bb863bbd24f602 WatchSource:0}: Error finding container cf4822e38064fa3ea2c40beae3522f947aa954c0dd3fe4b234bb863bbd24f602: Status 404 returned error can't find the container with id cf4822e38064fa3ea2c40beae3522f947aa954c0dd3fe4b234bb863bbd24f602
Oct 30 22:20:08.640073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.640035    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:08.640154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.640119    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:08.640192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:08.640162    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:09.048958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.048867    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" event={"ID":"ee0e8a04-1c10-4718-9e50-7816c5d5c35c","Type":"ContainerStarted","Data":"b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8"}
Oct 30 22:20:09.048958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.048933    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" event={"ID":"ee0e8a04-1c10-4718-9e50-7816c5d5c35c","Type":"ContainerStarted","Data":"f63740eaa4fda2e0def0e034e0fcd9e34a8d18c8c2a7d8fe6469a5c262a20ee3"}
Oct 30 22:20:09.051979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.051119    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" event={"ID":"887a1cf8-96c9-444f-a7cd-6847a0c28f74","Type":"ContainerStarted","Data":"2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a"}
Oct 30 22:20:09.051979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.051177    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" event={"ID":"887a1cf8-96c9-444f-a7cd-6847a0c28f74","Type":"ContainerStarted","Data":"cf4822e38064fa3ea2c40beae3522f947aa954c0dd3fe4b234bb863bbd24f602"}
Oct 30 22:20:09.102124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.102053    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podStartSLOduration=2.102037645 podStartE2EDuration="2.102037645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:09.091431024 +0000 UTC m=+793.130369913" watchObservedRunningTime="2026-10-30 22:20:09.102037645 +0000 UTC m=+793.140976531"
Oct 30 22:20:09.128001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.127931    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podStartSLOduration=2.127912689 podStartE2EDuration="2.127912689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:09.126937089 +0000 UTC m=+793.165875981" watchObservedRunningTime="2026-10-30 22:20:09.127912689 +0000 UTC m=+793.166851583"
Oct 30 22:20:09.308361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.304569    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:09.311109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.309722    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.311109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.309785    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:20:09.839039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.839006    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:09.840583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.840558    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.840723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:09.840604    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:20:10.303242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:10.302731    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:20:10.303242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:10.302815    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:20:10.834279 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:20:10.828963    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:20:10.838826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:10.838789    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:20:10.838989 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:10.838860    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:20:10.988245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:10.987754    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:11.015892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:11.015852    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:11.133948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:11.133911    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:20:11.307246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:11.305674    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.307246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:11.305733    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:20:11.838994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:11.838951    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.839244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:11.839014    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:20:12.303246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:12.302715    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.303246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:12.302786    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:20:12.839531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:12.839500    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.839776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:12.839755    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:20:13.303243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:13.302668    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.303243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:13.302729    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:20:13.839721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:13.839682    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.839993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:13.839968    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:20:14.305040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:14.304999    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.305809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:14.305775    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:20:14.838814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:14.838780    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.839004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:14.838837    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 22:20:15.303521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:15.303487    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:20:15.303752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:15.303540    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:20:15.843488 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:15.843458    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:15.843954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:15.843923    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:15.844745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:15.844722    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:20:16.307545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:16.307516    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:16.307988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:16.307959    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:16.308822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:16.308805    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:20:22.456126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.456079    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"]
Oct 30 22:20:22.462695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.462671    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.480815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.480780    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"]
Oct 30 22:20:22.517746 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.517714    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.518039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.518009    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.518128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.518046    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kll56\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-kube-api-access-kll56\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.518128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.518088    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.518208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.518159    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.518330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.518292    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.518433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.518416    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.518492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.518452    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.518546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.518513    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.619795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.619756    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.619807    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.619832    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kll56\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-kube-api-access-kll56\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.619865    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.619894    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.619953    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.620024    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.620049    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.620105    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.620350    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.620604    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.620785    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.620886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.620852    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.621005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.620980    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.622535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.622502    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.623128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.623109    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.639597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.639556    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kll56\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-kube-api-access-kll56\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.640028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.640004    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:22.778947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:22.778803    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:23.140588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:23.140555    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"]
Oct 30 22:20:23.141545 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:20:23.141516    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3bb4f9_d20b_4dc2_b0b9_702e9af23c2f.slice/crio-c61fee44e288ce72c45a2e234a72372e4f6f16230bd427abb8d695adb1118afa WatchSource:0}: Error finding container c61fee44e288ce72c45a2e234a72372e4f6f16230bd427abb8d695adb1118afa: Status 404 returned error can't find the container with id c61fee44e288ce72c45a2e234a72372e4f6f16230bd427abb8d695adb1118afa
Oct 30 22:20:23.144541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:23.144501    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:23.144637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:23.144584    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:23.144674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:23.144635    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:23.281351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:23.281302    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" event={"ID":"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f","Type":"ContainerStarted","Data":"a59bc00088cf63ff68d40159d80d9848ce12de85864c49fc1175bc0d25c5e667"}
Oct 30 22:20:23.281557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:23.281417    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" event={"ID":"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f","Type":"ContainerStarted","Data":"c61fee44e288ce72c45a2e234a72372e4f6f16230bd427abb8d695adb1118afa"}
Oct 30 22:20:23.324584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:23.324522    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podStartSLOduration=1.324505643 podStartE2EDuration="1.324505643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:23.319351964 +0000 UTC m=+807.358290849" watchObservedRunningTime="2026-10-30 22:20:23.324505643 +0000 UTC m=+807.363444529"
Oct 30 22:20:23.779247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:23.779087    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:24.781745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:24.781706    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:20:24.782171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:24.781777    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:20:25.780105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:25.779867    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:20:25.780105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:25.779956    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:20:26.782247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:26.779635    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:20:26.782247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:26.779700    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:20:27.779712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:27.779672    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:20:27.780036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:27.780004    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:20:28.781721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:28.781685    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:20:28.782195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:28.782169    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:20:29.779376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:29.779335    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:20:29.779567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:29.779404    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:20:30.476091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.476050    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"]
Oct 30 22:20:30.481790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.481763    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.503655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.503597    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"]
Oct 30 22:20:30.504043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.504015    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"distinct-2-w3-d52c59-kserve-self-signed-certs\""
Oct 30 22:20:30.653115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.653069    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-model-cache\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.653943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.653914    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfl5k\" (UniqueName: \"kubernetes.io/projected/337f84f2-d4e6-468a-9341-16b93e1b785b-kube-api-access-qfl5k\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.654198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.654179    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-dshm\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.654466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.654445    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337f84f2-d4e6-468a-9341-16b93e1b785b-tls-certs\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.654660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.654642    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-tmp-dir\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.654748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.654719    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-home\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.756037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.755931    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfl5k\" (UniqueName: \"kubernetes.io/projected/337f84f2-d4e6-468a-9341-16b93e1b785b-kube-api-access-qfl5k\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.756037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.756000    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-dshm\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.756283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.756067    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337f84f2-d4e6-468a-9341-16b93e1b785b-tls-certs\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.756283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.756247    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-tmp-dir\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.756283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.756277    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-home\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.756415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.756331    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-model-cache\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.756842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.756812    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-model-cache\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.756974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.756955    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-home\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.757626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.757605    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-tmp-dir\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.760804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.760768    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337f84f2-d4e6-468a-9341-16b93e1b785b-tls-certs\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.774120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.774087    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfl5k\" (UniqueName: \"kubernetes.io/projected/337f84f2-d4e6-468a-9341-16b93e1b785b-kube-api-access-qfl5k\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.778508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.778477    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-dshm\") pod \"distinct-2-w3-d52c59-kserve-5756975d7-ddl2c\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:30.786157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.786131    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:30.786492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.786464    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:30.787499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.787477    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:20:30.809472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:30.809440    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:31.174577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:31.174553    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"]
Oct 30 22:20:31.176797 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:20:31.176771    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod337f84f2_d4e6_468a_9341_16b93e1b785b.slice/crio-2e80af4bfa60288db300d33918fdc57921cdb6d9d30acf9e87e72f0a881628f0 WatchSource:0}: Error finding container 2e80af4bfa60288db300d33918fdc57921cdb6d9d30acf9e87e72f0a881628f0: Status 404 returned error can't find the container with id 2e80af4bfa60288db300d33918fdc57921cdb6d9d30acf9e87e72f0a881628f0
Oct 30 22:20:31.356862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:31.356746    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" event={"ID":"337f84f2-d4e6-468a-9341-16b93e1b785b","Type":"ContainerStarted","Data":"7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb"}
Oct 30 22:20:31.356862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:31.356797    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" event={"ID":"337f84f2-d4e6-468a-9341-16b93e1b785b","Type":"ContainerStarted","Data":"2e80af4bfa60288db300d33918fdc57921cdb6d9d30acf9e87e72f0a881628f0"}
Oct 30 22:20:31.393871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:31.393533    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podStartSLOduration=1.393512804 podStartE2EDuration="1.393512804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:31.390809546 +0000 UTC m=+815.429748432" watchObservedRunningTime="2026-10-30 22:20:31.393512804 +0000 UTC m=+815.432451687"
Oct 30 22:20:31.810540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:31.810500    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:37.338678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.338640    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"]
Oct 30 22:20:37.344035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.344008    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.373368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.373334    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"]
Oct 30 22:20:37.469700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.469661    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.469914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.469732    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.469914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.469834    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.469914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.469879    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmrk\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-kube-api-access-hfmrk\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.470079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.469973    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.470079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.470023    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.470079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.470048    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.470266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.470165    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.470266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.470202    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571378    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571454    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571480    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571509    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmrk\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-kube-api-access-hfmrk\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571543    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571576    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571604    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571675    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.571984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.571719    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.572112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.572086    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.572174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.572110    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.572174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.572154    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.572360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.572341    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.572420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.572366    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.574307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.574275    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.574471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.574455    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.588621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.588597    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.588890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.588847    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmrk\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-kube-api-access-hfmrk\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.660313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.660279    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:37.985993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.985971    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"]
Oct 30 22:20:37.987792 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:20:37.987762    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c958d0c_10a1_4b47_a326_e9b877ce1f68.slice/crio-ca2430dc6b431467de34e77f81d4c05e9d8aef418886a0620915af912e6a4f18 WatchSource:0}: Error finding container ca2430dc6b431467de34e77f81d4c05e9d8aef418886a0620915af912e6a4f18: Status 404 returned error can't find the container with id ca2430dc6b431467de34e77f81d4c05e9d8aef418886a0620915af912e6a4f18
Oct 30 22:20:37.990901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.990863    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:37.991036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.990947    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:37.991036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:37.990997    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:38.425959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:38.425913    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" event={"ID":"3c958d0c-10a1-4b47-a326-e9b877ce1f68","Type":"ContainerStarted","Data":"20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2"}
Oct 30 22:20:38.426494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:38.425982    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" event={"ID":"3c958d0c-10a1-4b47-a326-e9b877ce1f68","Type":"ContainerStarted","Data":"ca2430dc6b431467de34e77f81d4c05e9d8aef418886a0620915af912e6a4f18"}
Oct 30 22:20:38.476163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:38.476109    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" podStartSLOduration=1.476094151 podStartE2EDuration="1.476094151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:38.471836409 +0000 UTC m=+822.510775295" watchObservedRunningTime="2026-10-30 22:20:38.476094151 +0000 UTC m=+822.515033124"
Oct 30 22:20:38.663243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:38.662408    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:39.509646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.509609    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"]
Oct 30 22:20:39.513982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.513957    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.537273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.537237    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"]
Oct 30 22:20:39.538657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.538632    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"default-dockercfg-pm7m5\""
Oct 30 22:20:39.538910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.538879    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"distinct-w1-2b1e88-kserve-self-signed-certs\""
Oct 30 22:20:39.578523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.578490    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"openshift-service-ca.crt\""
Oct 30 22:20:39.578895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.578873    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"kube-root-ca.crt\""
Oct 30 22:20:39.599601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.599564    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmzh\" (UniqueName: \"kubernetes.io/projected/a9304141-f4c1-4196-8ffc-acbeb31960f7-kube-api-access-cgmzh\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.599774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.599732    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-tmp-dir\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.599817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.599785    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9304141-f4c1-4196-8ffc-acbeb31960f7-tls-certs\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.599817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.599806    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-model-cache\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.599932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.599918    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-dshm\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.600025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.600010    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-home\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.669265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.669199    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:20:39.669578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.669541    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:20:39.701372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.701171    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-home\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.701372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.701270    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmzh\" (UniqueName: \"kubernetes.io/projected/a9304141-f4c1-4196-8ffc-acbeb31960f7-kube-api-access-cgmzh\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.703163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.701444    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-tmp-dir\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.703163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.701520    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9304141-f4c1-4196-8ffc-acbeb31960f7-tls-certs\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.703163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.701554    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-model-cache\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.703163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.701620    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-dshm\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.703163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.702439    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-tmp-dir\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.703163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.703118    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-home\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.704968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.703276    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-model-cache\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.705075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.705024    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-dshm\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.730884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.715547    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9304141-f4c1-4196-8ffc-acbeb31960f7-tls-certs\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.778659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.777903    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmzh\" (UniqueName: \"kubernetes.io/projected/a9304141-f4c1-4196-8ffc-acbeb31960f7-kube-api-access-cgmzh\") pod \"distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") " pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:39.830300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:39.830265    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:40.183397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.183365    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"]
Oct 30 22:20:40.185050 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:20:40.185024    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9304141_f4c1_4196_8ffc_acbeb31960f7.slice/crio-17b3a4b22df56fbb0401d58bd3bea7eff90679e3400c674513aedf98f3305f3b WatchSource:0}: Error finding container 17b3a4b22df56fbb0401d58bd3bea7eff90679e3400c674513aedf98f3305f3b: Status 404 returned error can't find the container with id 17b3a4b22df56fbb0401d58bd3bea7eff90679e3400c674513aedf98f3305f3b
Oct 30 22:20:40.445508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.445415    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" event={"ID":"a9304141-f4c1-4196-8ffc-acbeb31960f7","Type":"ContainerStarted","Data":"e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a"}
Oct 30 22:20:40.445508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.445467    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" event={"ID":"a9304141-f4c1-4196-8ffc-acbeb31960f7","Type":"ContainerStarted","Data":"17b3a4b22df56fbb0401d58bd3bea7eff90679e3400c674513aedf98f3305f3b"}
Oct 30 22:20:40.486147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.486086    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podStartSLOduration=1.486071076 podStartE2EDuration="1.486071076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:40.482932706 +0000 UTC m=+824.521871594" watchObservedRunningTime="2026-10-30 22:20:40.486071076 +0000 UTC m=+824.525009962"
Oct 30 22:20:40.662646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.662603    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:20:40.663161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.662665    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:20:40.809837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.809739    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:40.821514 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:20:40.821475    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:20:40.828987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.828959    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:40.831105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:40.831085    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:41.489274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:41.489206    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:20:41.674243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:41.663634    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:20:41.674243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:41.663696    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:20:42.662669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:42.662628    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:20:42.662959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:42.662927    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:20:43.673242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:43.672651    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:20:43.673242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:43.672721    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:20:44.660977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:44.660940    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:20:44.661161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:44.660995    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:20:45.665873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:45.665839    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:45.666512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:45.666318    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:45.667075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:45.667055    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:20:49.830758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:49.830716    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:49.843759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:49.843732    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:50.626708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:50.626658    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:20:52.953493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:52.953454    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"]
Oct 30 22:20:52.960238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:52.960189    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:52.994662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:52.994628    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"]
Oct 30 22:20:53.065114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065065    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.065342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065177    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.065342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065272    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.065342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065313    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.065522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065392    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.065522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065423    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.065522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065482    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.065843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065527    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsrc\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-kube-api-access-dcsrc\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.065843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.065615    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.166755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.166711    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsrc\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-kube-api-access-dcsrc\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.166973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.166817    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.166973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.166882    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.166973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.166946    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167003    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167042    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167122    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167152    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167238    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167565    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167595    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167699    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.167825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167803    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.168139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.167854    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.170308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.170281    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.170551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.170524    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.184321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.184293    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.185458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.185426    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsrc\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-kube-api-access-dcsrc\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.304868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.304776    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:53.644951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.644913    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"]
Oct 30 22:20:53.646395 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:20:53.646367    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b8345e_b2e7_48e1_a0f3_b02bf0e0af13.slice/crio-b27c61f2ad142b1ee9940d7c01ff1b92400a4d5cd717e1090fab303d1312290b WatchSource:0}: Error finding container b27c61f2ad142b1ee9940d7c01ff1b92400a4d5cd717e1090fab303d1312290b: Status 404 returned error can't find the container with id b27c61f2ad142b1ee9940d7c01ff1b92400a4d5cd717e1090fab303d1312290b
Oct 30 22:20:53.649734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.649688    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:53.649860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.649785    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:53.649860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.649827    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:20:53.670557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:53.670484    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" event={"ID":"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13","Type":"ContainerStarted","Data":"b27c61f2ad142b1ee9940d7c01ff1b92400a4d5cd717e1090fab303d1312290b"}
Oct 30 22:20:54.693688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:54.693647    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" event={"ID":"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13","Type":"ContainerStarted","Data":"b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7"}
Oct 30 22:20:54.744610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:54.744552    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podStartSLOduration=2.744536067 podStartE2EDuration="2.744536067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:54.740382464 +0000 UTC m=+838.779321350" watchObservedRunningTime="2026-10-30 22:20:54.744536067 +0000 UTC m=+838.783474952"
Oct 30 22:20:55.305198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:55.305158    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:20:55.307092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:55.307066    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:20:55.307277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:55.307126    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 22:20:56.305754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:56.305711    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:20:56.306263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:56.305768    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 22:20:57.305820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:57.305772    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:20:57.306254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:57.305845    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 22:20:58.306273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:58.306194    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:20:58.306690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:58.306291    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 22:20:59.305602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:59.305561    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:20:59.305779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:20:59.305631    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 22:21:00.305717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:00.305671    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:21:00.306317 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:00.305737    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 22:21:01.311171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:01.311136    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:21:01.311735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:01.311718    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:21:01.312567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:01.312544    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:21:10.820711 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:21:10.820614    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:21:20.311476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.311430    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"]
Oct 30 22:21:20.316926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.316897    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.333455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.333428    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"]
Oct 30 22:21:20.336870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.336839    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:20.337012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.336903    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:20.337012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.336929    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:20.338588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.338536    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:21:20.401134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.401096    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.401352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.401153    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/4087e96b-4767-4fb0-a808-fee382dedb69-kube-api-access-55h2j\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.401352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.401281    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.502734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.502694    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.502959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.502934    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/4087e96b-4767-4fb0-a808-fee382dedb69-kube-api-access-55h2j\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.503042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.502993    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.503862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.503831    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.505871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.505839    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.519376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.519343    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/4087e96b-4767-4fb0-a808-fee382dedb69-kube-api-access-55h2j\") pod \"maas-api-revoke-sub-d6616045-npdqg\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.631668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.631633    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:20.955604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:20.955576    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"]
Oct 30 22:21:20.957562 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:21:20.957535    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4087e96b_4767_4fb0_a808_fee382dedb69.slice/crio-815e9e217bbad173a653388d1ffd507add3e1eb26308a8349915670c37fa4cbb WatchSource:0}: Error finding container 815e9e217bbad173a653388d1ffd507add3e1eb26308a8349915670c37fa4cbb: Status 404 returned error can't find the container with id 815e9e217bbad173a653388d1ffd507add3e1eb26308a8349915670c37fa4cbb
Oct 30 22:21:21.040333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:21.040295    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg" event={"ID":"4087e96b-4767-4fb0-a808-fee382dedb69","Type":"ContainerStarted","Data":"815e9e217bbad173a653388d1ffd507add3e1eb26308a8349915670c37fa4cbb"}
Oct 30 22:21:22.048993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.048956    4581 generic.go:356] "Generic (PLEG): container finished" podID="4087e96b-4767-4fb0-a808-fee382dedb69" containerID="6c5320f783df0e9b417ec9cbdfcd4cf190bd59f10f62b3ea62b15092db7b97ac" exitCode=0
Oct 30 22:21:22.049427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.049042    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg" event={"ID":"4087e96b-4767-4fb0-a808-fee382dedb69","Type":"ContainerDied","Data":"6c5320f783df0e9b417ec9cbdfcd4cf190bd59f10f62b3ea62b15092db7b97ac"}
Oct 30 22:21:22.615427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.615392    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"]
Oct 30 22:21:22.620297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.620273    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.657655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.652356    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"]
Oct 30 22:21:22.738808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.738769    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5467bf2c-3a86-4dc1-9975-381cbb821574-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.739260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.738875    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.739260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.738927    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.739260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.739048    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.739260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.739086    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.739260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.739179    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.739567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.739277    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.739567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.739301    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2p74\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-kube-api-access-h2p74\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.739567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.739394    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.840855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.840819    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.840855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.840857    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.840949    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.840992    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841029    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841045    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841064    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p74\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-kube-api-access-h2p74\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841118    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841242    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5467bf2c-3a86-4dc1-9975-381cbb821574-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841324    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841541    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841666    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841706    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.841992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.841965    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5467bf2c-3a86-4dc1-9975-381cbb821574-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.843570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.843550    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.844158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.844136    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.858305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.858273    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.858990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.858966    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p74\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-kube-api-access-h2p74\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:22.936096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:22.936016    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:23.191705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.191612    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"]
Oct 30 22:21:23.211483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.211448    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.223183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.223154    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"]
Oct 30 22:21:23.235559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.235530    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:23.275698 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:21:23.275670    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5467bf2c_3a86_4dc1_9975_381cbb821574.slice/crio-31bced79189145adf5efb30716b8e14765f07ae7627cc2c3b61ef6dfb5593c10 WatchSource:0}: Error finding container 31bced79189145adf5efb30716b8e14765f07ae7627cc2c3b61ef6dfb5593c10: Status 404 returned error can't find the container with id 31bced79189145adf5efb30716b8e14765f07ae7627cc2c3b61ef6dfb5593c10
Oct 30 22:21:23.275698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.275677    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"]
Oct 30 22:21:23.278817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.278774    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:23.278959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.278934    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:23.279035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.279009    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:23.347756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.347719    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4087e96b-4767-4fb0-a808-fee382dedb69-kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/4087e96b-4767-4fb0-a808-fee382dedb69-kube-api-access-55h2j\") pod \"4087e96b-4767-4fb0-a808-fee382dedb69\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") "
Oct 30 22:21:23.347983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.347966    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-xks-ca\") pod \"4087e96b-4767-4fb0-a808-fee382dedb69\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") "
Oct 30 22:21:23.348136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348117    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-openshift-ca\") pod \"4087e96b-4767-4fb0-a808-fee382dedb69\" (UID: \"4087e96b-4767-4fb0-a808-fee382dedb69\") "
Oct 30 22:21:23.348484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348460    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-openshift-ca" pod "4087e96b-4767-4fb0-a808-fee382dedb69" (UID: "4087e96b-4767-4fb0-a808-fee382dedb69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:23.348546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348495    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.348582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348554    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.348625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348602    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.348707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348692    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.348768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348757    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.348810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348797    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.348863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348852    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.348900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348875    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2f5v\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-kube-api-access-j2f5v\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.348933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.348914    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.349010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.349000    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:23.350559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.350533    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4087e96b-4767-4fb0-a808-fee382dedb69-kube-api-access-55h2j" pod "4087e96b-4767-4fb0-a808-fee382dedb69" (UID: "4087e96b-4767-4fb0-a808-fee382dedb69"). InnerVolumeSpecName "kube-api-access-55h2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:23.350927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.350903    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-xks-ca" pod "4087e96b-4767-4fb0-a808-fee382dedb69" (UID: "4087e96b-4767-4fb0-a808-fee382dedb69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:23.450390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.450290    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.450390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.450364    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.450620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.450391    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.450686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.450624    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.450984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.450955    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.451082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451007    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.451082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451014    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.451933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451445    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.451933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451539    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.451933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451580    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.451933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451635    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2f5v\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-kube-api-access-j2f5v\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.451933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451677    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.451933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451715    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.452338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451953    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4087e96b-4767-4fb0-a808-fee382dedb69-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:23.452338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.451974    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/4087e96b-4767-4fb0-a808-fee382dedb69-kube-api-access-55h2j\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:23.452441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.452364    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.453441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.453418    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.454521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.454502    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.469108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.469085    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.469739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.469706    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2f5v\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-kube-api-access-j2f5v\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.537105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.537070    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:23.876274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.876208    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"]
Oct 30 22:21:23.877235 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:21:23.877186    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dda98e_6ca9_4944_8d0d_1f9a9380835a.slice/crio-af387ea80aa9a821878e1ce9e61e50f1d147733eb93e8f9bf7a0b3567217d1ab WatchSource:0}: Error finding container af387ea80aa9a821878e1ce9e61e50f1d147733eb93e8f9bf7a0b3567217d1ab: Status 404 returned error can't find the container with id af387ea80aa9a821878e1ce9e61e50f1d147733eb93e8f9bf7a0b3567217d1ab
Oct 30 22:21:23.880325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.880289    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:23.880455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.880377    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:23.880455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:23.880422    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:24.075763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.075732    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"
Oct 30 22:21:24.075959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.075760    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg" event={"ID":"4087e96b-4767-4fb0-a808-fee382dedb69","Type":"ContainerDied","Data":"815e9e217bbad173a653388d1ffd507add3e1eb26308a8349915670c37fa4cbb"}
Oct 30 22:21:24.075959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.075826    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="815e9e217bbad173a653388d1ffd507add3e1eb26308a8349915670c37fa4cbb"
Oct 30 22:21:24.079372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.078251    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" event={"ID":"d5dda98e-6ca9-4944-8d0d-1f9a9380835a","Type":"ContainerStarted","Data":"d6b2f60034a05403bff29c2182833d2b3f50aa45df9f0aa9004be58d4c9735a4"}
Oct 30 22:21:24.079372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.078306    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" event={"ID":"d5dda98e-6ca9-4944-8d0d-1f9a9380835a","Type":"ContainerStarted","Data":"af387ea80aa9a821878e1ce9e61e50f1d147733eb93e8f9bf7a0b3567217d1ab"}
Oct 30 22:21:24.105605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.105570    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" event={"ID":"5467bf2c-3a86-4dc1-9975-381cbb821574","Type":"ContainerStarted","Data":"03ce93aa199cc0c0a462fc51b9996e0fe14615921ca7e960134b5ef94e7db054"}
Oct 30 22:21:24.105766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.105631    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" event={"ID":"5467bf2c-3a86-4dc1-9975-381cbb821574","Type":"ContainerStarted","Data":"31bced79189145adf5efb30716b8e14765f07ae7627cc2c3b61ef6dfb5593c10"}
Oct 30 22:21:24.129318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.129049    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podStartSLOduration=1.129029949 podStartE2EDuration="1.129029949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:24.125372589 +0000 UTC m=+868.164311476" watchObservedRunningTime="2026-10-30 22:21:24.129029949 +0000 UTC m=+868.167968838"
Oct 30 22:21:24.215326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.214069    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" podStartSLOduration=2.214043773 podStartE2EDuration="2.214043773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:24.162987183 +0000 UTC m=+868.201926071" watchObservedRunningTime="2026-10-30 22:21:24.214043773 +0000 UTC m=+868.252982661"
Oct 30 22:21:24.538129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.538031    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:24.539683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.539656    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:21:24.539813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.539709    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 22:21:24.940712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.940397    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:24.940712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.940588    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:21:24.940712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:24.940636    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 22:21:25.537441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:25.537407    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:21:25.537911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:25.537886    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 22:21:25.936505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:25.936458    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:21:25.936715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:25.936542    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 22:21:26.537679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:26.537640    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:21:26.538093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:26.537738    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 22:21:26.936967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:26.936923    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:21:26.937154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:26.937004    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 22:21:27.537444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:27.537411    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:21:27.537650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:27.537486    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 22:21:27.937000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:27.936967    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:21:27.937484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:27.937042    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 22:21:28.538250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:28.538189    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:21:28.538491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:28.538309    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 22:21:28.937370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:28.937336    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:21:28.937764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:28.937407    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 22:21:29.537651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:29.537616    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:21:29.537843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:29.537672    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 22:21:29.936680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:29.936631    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:21:29.936913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:29.936685    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 22:21:30.537764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:30.537728    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:21:30.538314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:30.537784    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 22:21:31.506496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:31.506410    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:31.518250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:31.508947    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:31.518250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:31.511241    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:21:31.880966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:31.880932    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:31.882464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:31.882404    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:31.883752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:31.883725    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:21:32.871460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:32.871407    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"]
Oct 30 22:21:32.872921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:32.872893    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4087e96b-4767-4fb0-a808-fee382dedb69" containerName="revoke-keys"
Oct 30 22:21:32.873073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:32.873061    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4087e96b-4767-4fb0-a808-fee382dedb69" containerName="revoke-keys"
Oct 30 22:21:32.873409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:32.873391    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4087e96b-4767-4fb0-a808-fee382dedb69" containerName="revoke-keys"
Oct 30 22:21:32.904612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:32.904575    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:32.946978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:32.945607    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"distinct-2-w5-54114e-kserve-self-signed-certs\""
Oct 30 22:21:33.033730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.033678    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-home\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.034029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.033997    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tmp-dir\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.034167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.034153    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzc2g\" (UniqueName: \"kubernetes.io/projected/9f861b1e-34e1-4f94-8074-f0cb46980b9c-kube-api-access-xzc2g\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.034352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.034338    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-model-cache\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.034514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.034490    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tls-certs\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.034774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.034761    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-dshm\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.162371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.158354    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tls-certs\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.162371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.158495    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-dshm\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.162371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.158731    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-home\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.162371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.158788    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tmp-dir\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.162371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.158832    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzc2g\" (UniqueName: \"kubernetes.io/projected/9f861b1e-34e1-4f94-8074-f0cb46980b9c-kube-api-access-xzc2g\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.162371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.158933    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-model-cache\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.162371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.159464    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-model-cache\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.174242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.163744    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tmp-dir\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.237775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.237734    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-home\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.286800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.286724    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"]
Oct 30 22:21:33.307856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.306183    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-dshm\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.309014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.308986    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tls-certs\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.425114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.425027    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzc2g\" (UniqueName: \"kubernetes.io/projected/9f861b1e-34e1-4f94-8074-f0cb46980b9c-kube-api-access-xzc2g\") pod \"distinct-2-w5-54114e-kserve-6899fc7795-65rhs\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") " pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.479080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.477325    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:33.652243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.651519    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:33.683661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.683557    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:33.797333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.797286    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"]
Oct 30 22:21:33.856287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.830171    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"]
Oct 30 22:21:33.856838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.856686    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:33.882263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.870394    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:33.882263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.870541    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:33.919497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.919465    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:21:33.920083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.920062    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:33.957501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.948680    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn8qm\" (UniqueName: \"kubernetes.io/projected/106ff678-a388-4db1-8947-d5c834a140c7-kube-api-access-wn8qm\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:33.957501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.948808    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:33.957501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:33.948917    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:34.058257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:34.055100    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:34.058257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:34.055296    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:34.058257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:34.055424    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn8qm\" (UniqueName: \"kubernetes.io/projected/106ff678-a388-4db1-8947-d5c834a140c7-kube-api-access-wn8qm\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:34.058257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:34.057306    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:34.078388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:34.078347    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:34.183682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:34.183628    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn8qm\" (UniqueName: \"kubernetes.io/projected/106ff678-a388-4db1-8947-d5c834a140c7-kube-api-access-wn8qm\") pod \"maas-api-revoke-sub-c6c5366d-cbcpg\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:34.335029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:34.334938    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:34.662395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:34.662351    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:35.193819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:35.193772    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"]
Oct 30 22:21:35.406567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:35.406512    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"]
Oct 30 22:21:35.844417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:35.844376    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" event={"ID":"9f861b1e-34e1-4f94-8074-f0cb46980b9c","Type":"ContainerStarted","Data":"9d18f5ca0fd33b3b6b51b3f2f36df09d8c0e6ad627de06bc86149649067bb5ec"}
Oct 30 22:21:35.844609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:35.844426    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" event={"ID":"9f861b1e-34e1-4f94-8074-f0cb46980b9c","Type":"ContainerStarted","Data":"49a40147359bc30b17dedb31d6837ae822c0a5b94e8b6974369ba63431d3b0ff"}
Oct 30 22:21:35.846095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:35.846066    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg" event={"ID":"106ff678-a388-4db1-8947-d5c834a140c7","Type":"ContainerStarted","Data":"b99620063d7009e73bdc2a37905b440a764cb939277bf0ea527825e4420ebd31"}
Oct 30 22:21:35.846255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:35.846124    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg" event={"ID":"106ff678-a388-4db1-8947-d5c834a140c7","Type":"ContainerStarted","Data":"3ec4dd61598143fa83a68bbe2c90ea5b23d41daaf8ed9aeb6eb49fb8ab4f9016"}
Oct 30 22:21:35.882662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:35.882593    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podStartSLOduration=3.88257452 podStartE2EDuration="3.88257452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:35.877506937 +0000 UTC m=+879.916445824" watchObservedRunningTime="2026-10-30 22:21:35.88257452 +0000 UTC m=+879.921513406"
Oct 30 22:21:35.915556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:35.915494    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg" podStartSLOduration=2.915474535 podStartE2EDuration="2.915474535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:35.91275646 +0000 UTC m=+879.951695345" watchObservedRunningTime="2026-10-30 22:21:35.915474535 +0000 UTC m=+879.954413424"
Oct 30 22:21:36.689019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:36.688960    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:36.856749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:36.856711    4581 generic.go:356] "Generic (PLEG): container finished" podID="106ff678-a388-4db1-8947-d5c834a140c7" containerID="b99620063d7009e73bdc2a37905b440a764cb939277bf0ea527825e4420ebd31" exitCode=0
Oct 30 22:21:36.856934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:36.856799    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg" event={"ID":"106ff678-a388-4db1-8947-d5c834a140c7","Type":"ContainerDied","Data":"b99620063d7009e73bdc2a37905b440a764cb939277bf0ea527825e4420ebd31"}
Oct 30 22:21:37.916289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.916192    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"]
Oct 30 22:21:37.921068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.921038    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.937012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.936978    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603e680b-f3b7-425b-83ac-b7e7051cd801-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.937330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.937305    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.937609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.937584    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.937710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.937626    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwsbj\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-kube-api-access-xwsbj\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.937710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.937677    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.938017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.937996    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.938111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.938097    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.938151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.938125    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.938197 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.938183    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:37.966597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:37.966556    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"]
Oct 30 22:21:38.039545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039510    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603e680b-f3b7-425b-83ac-b7e7051cd801-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.039730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039588    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.039730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039700    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.039853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039734    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwsbj\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-kube-api-access-xwsbj\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.039853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039787    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.039853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039815    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.040011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039863    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.040011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039893    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.040011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.039931    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.040161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.040027    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.040255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.040232    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603e680b-f3b7-425b-83ac-b7e7051cd801-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.040350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.040331    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.040413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.040393    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.040757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.040735    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.042655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.042631    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.043737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.043715    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.061700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.061673    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.061905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.061879    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwsbj\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-kube-api-access-xwsbj\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.078963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.078931    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"]
Oct 30 22:21:38.097318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.095330    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.120309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.117574    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"]
Oct 30 22:21:38.121240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.121148    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.149387    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-xks-ca\") pod \"106ff678-a388-4db1-8947-d5c834a140c7\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") "
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.149450    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-openshift-ca\") pod \"106ff678-a388-4db1-8947-d5c834a140c7\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") "
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.149528    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106ff678-a388-4db1-8947-d5c834a140c7-kube-api-access-wn8qm\" (UniqueName: \"kubernetes.io/projected/106ff678-a388-4db1-8947-d5c834a140c7-kube-api-access-wn8qm\") pod \"106ff678-a388-4db1-8947-d5c834a140c7\" (UID: \"106ff678-a388-4db1-8947-d5c834a140c7\") "
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150184    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150237    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150295    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150364    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150448    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150484    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150612    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfqc\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-kube-api-access-6sfqc\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150733    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.152260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.150808    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.155974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.155938    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-openshift-ca" pod "106ff678-a388-4db1-8947-d5c834a140c7" (UID: "106ff678-a388-4db1-8947-d5c834a140c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:38.159517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.159482    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-xks-ca" pod "106ff678-a388-4db1-8947-d5c834a140c7" (UID: "106ff678-a388-4db1-8947-d5c834a140c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:38.159743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.159720    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106ff678-a388-4db1-8947-d5c834a140c7-kube-api-access-wn8qm" pod "106ff678-a388-4db1-8947-d5c834a140c7" (UID: "106ff678-a388-4db1-8947-d5c834a140c7"). InnerVolumeSpecName "kube-api-access-wn8qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:38.239979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.239949    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:38.252627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252592    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.252758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252672    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.252758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252734    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.252857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252759    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.252857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252808    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.252857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252851    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.253021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252919    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.253021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252952    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.253021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.252975    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.253130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.253063    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfqc\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-kube-api-access-6sfqc\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.253167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.253144    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:38.253167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.253158    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106ff678-a388-4db1-8947-d5c834a140c7-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:38.253271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.253167    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn8qm\" (UniqueName: \"kubernetes.io/projected/106ff678-a388-4db1-8947-d5c834a140c7-kube-api-access-wn8qm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:38.253845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.253815    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.253963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.253816    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.254075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.254048    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.254300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.254281    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.256009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.255977    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.256698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.256676    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.270441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.270408    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.270762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.270728    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfqc\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-kube-api-access-6sfqc\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.429166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.429064    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:38.577118 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:21:38.577084    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603e680b_f3b7_425b_83ac_b7e7051cd801.slice/crio-32f90b1398e4dc174d51e1391b38e0c8269932a71f58bf71ace38b9dfcd6ae25 WatchSource:0}: Error finding container 32f90b1398e4dc174d51e1391b38e0c8269932a71f58bf71ace38b9dfcd6ae25: Status 404 returned error can't find the container with id 32f90b1398e4dc174d51e1391b38e0c8269932a71f58bf71ace38b9dfcd6ae25
Oct 30 22:21:38.578001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.577973    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"]
Oct 30 22:21:38.581502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.581466    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:38.581626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.581536    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:38.581626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.581565    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:38.755024 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.754992    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"]
Oct 30 22:21:38.755516 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:21:38.755487    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d3182d_81a5_4591_9a6d_ce1bfffeb438.slice/crio-2a2e766ca17b034b87f0edb1c546ec02a5dd403e1c78f71922fbaae08a2d5460 WatchSource:0}: Error finding container 2a2e766ca17b034b87f0edb1c546ec02a5dd403e1c78f71922fbaae08a2d5460: Status 404 returned error can't find the container with id 2a2e766ca17b034b87f0edb1c546ec02a5dd403e1c78f71922fbaae08a2d5460
Oct 30 22:21:38.758557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.758518    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:38.758665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.758588    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:38.758665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.758618    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:38.894579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.894029    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" event={"ID":"74d3182d-81a5-4591-9a6d-ce1bfffeb438","Type":"ContainerStarted","Data":"afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27"}
Oct 30 22:21:38.894579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.894097    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" event={"ID":"74d3182d-81a5-4591-9a6d-ce1bfffeb438","Type":"ContainerStarted","Data":"2a2e766ca17b034b87f0edb1c546ec02a5dd403e1c78f71922fbaae08a2d5460"}
Oct 30 22:21:38.899740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.898755    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" event={"ID":"603e680b-f3b7-425b-83ac-b7e7051cd801","Type":"ContainerStarted","Data":"762af472bc458080588b7192c41771ae4a4619665e2b6a0db4fa47e764bbcc13"}
Oct 30 22:21:38.899740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.898847    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" event={"ID":"603e680b-f3b7-425b-83ac-b7e7051cd801","Type":"ContainerStarted","Data":"32f90b1398e4dc174d51e1391b38e0c8269932a71f58bf71ace38b9dfcd6ae25"}
Oct 30 22:21:38.899740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.898977    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg" event={"ID":"106ff678-a388-4db1-8947-d5c834a140c7","Type":"ContainerDied","Data":"3ec4dd61598143fa83a68bbe2c90ea5b23d41daaf8ed9aeb6eb49fb8ab4f9016"}
Oct 30 22:21:38.899740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.899019    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec4dd61598143fa83a68bbe2c90ea5b23d41daaf8ed9aeb6eb49fb8ab4f9016"
Oct 30 22:21:38.899740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.899067    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"
Oct 30 22:21:38.934574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:38.934517    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podStartSLOduration=0.934504706 podStartE2EDuration="934.504706ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:38.93380192 +0000 UTC m=+882.972740806" watchObservedRunningTime="2026-10-30 22:21:38.934504706 +0000 UTC m=+882.973443592"
Oct 30 22:21:39.000104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:39.000033    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podStartSLOduration=2.000011655 podStartE2EDuration="2.000011655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:38.993322748 +0000 UTC m=+883.032261640" watchObservedRunningTime="2026-10-30 22:21:39.000011655 +0000 UTC m=+883.038950565"
Oct 30 22:21:39.241150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:39.241118    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:39.242544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:39.242515    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:21:39.242689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:39.242580    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:21:39.429756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:39.429560    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:39.431045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:39.430931    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:21:39.431045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:39.430998    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:21:40.241341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:40.241207    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.241341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:40.241312    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:21:40.429849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:40.429811    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.430012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:40.429893    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:21:40.843024 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:21:40.842980    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:21:41.240579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:41.240544    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.240779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:41.240605    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:21:41.430384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:41.429988    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.430384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:41.430041    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:21:42.241304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:42.241266    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.241469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:42.241328    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:21:42.429464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:42.429416    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.429644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:42.429483    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:21:43.241093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.240917    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.241093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.241004    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:21:43.432243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.430631    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.432243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.430697    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:21:43.466036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.465874    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"]
Oct 30 22:21:43.467003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.466969    4581 cpu_manager.go:414] "containerMap: removing container" podUID="106ff678-a388-4db1-8947-d5c834a140c7" containerName="revoke-keys"
Oct 30 22:21:43.467003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.467002    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106ff678-a388-4db1-8947-d5c834a140c7" containerName="revoke-keys"
Oct 30 22:21:43.467257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.467162    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="106ff678-a388-4db1-8947-d5c834a140c7" containerName="revoke-keys"
Oct 30 22:21:43.616510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.616473    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"]
Oct 30 22:21:43.616865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.616846    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.640298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.635928    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:21:43.647733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.645667    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:43.647733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.645717    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:43.647733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.645822    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:43.714319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.714282    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:43.718449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.718414    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mdd\" (UniqueName: \"kubernetes.io/projected/9be5f1ed-bc46-45cb-a778-bbb6e57db950-kube-api-access-s8mdd\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.718749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.718725    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.719154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.719132    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.731470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.731186    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:43.821073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.821036    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.821344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.821091    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mdd\" (UniqueName: \"kubernetes.io/projected/9be5f1ed-bc46-45cb-a778-bbb6e57db950-kube-api-access-s8mdd\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.821344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.821149    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.821845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.821820    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.823751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.823725    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:43.838449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:43.838424    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mdd\" (UniqueName: \"kubernetes.io/projected/9be5f1ed-bc46-45cb-a778-bbb6e57db950-kube-api-access-s8mdd\") pod \"maas-api-revoke-sub-e8d20ee0-z65zm\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:44.016181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.016083    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:44.023807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.023769    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:21:44.240623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.240571    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.240928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.240886    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:21:44.430255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.429577    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.430255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.429658    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:21:44.545299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.545266    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"]
Oct 30 22:21:44.623248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.617311    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"]
Oct 30 22:21:44.632794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.632771    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.634736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.634687    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"]
Oct 30 22:21:44.748669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.748633    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k4zx\" (UniqueName: \"kubernetes.io/projected/bf94f1e0-8df4-409d-adcb-7d0423161a7f-kube-api-access-2k4zx\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.748851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.748820    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.748913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.748896    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.850307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.850175    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.858246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.854393    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.858246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.854513    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4zx\" (UniqueName: \"kubernetes.io/projected/bf94f1e0-8df4-409d-adcb-7d0423161a7f-kube-api-access-2k4zx\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.870268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.866054    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.878242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.877633    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:44.907307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:44.905250    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4zx\" (UniqueName: \"kubernetes.io/projected/bf94f1e0-8df4-409d-adcb-7d0423161a7f-kube-api-access-2k4zx\") pod \"maas-api-revoke-sub-86bee085-8crdg\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:45.013898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:45.013861    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm" event={"ID":"9be5f1ed-bc46-45cb-a778-bbb6e57db950","Type":"ContainerStarted","Data":"1ba48d77d314ee5c43e106b6cafb5e683fe5f9526981f86cfda71317a6d22ad3"}
Oct 30 22:21:45.014554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:45.014538    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:45.240667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:45.240635    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:21:45.240785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:45.240693    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:21:45.429762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:45.429671    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:21:45.429762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:45.429726    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 22:21:45.451243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:45.451173    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"]
Oct 30 22:21:46.027762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.027651    4581 generic.go:356] "Generic (PLEG): container finished" podID="9be5f1ed-bc46-45cb-a778-bbb6e57db950" containerID="566c99d089d40d6686cfc4fb606464ba23d6c7854e559fd952cec017b16c2c06" exitCode=0
Oct 30 22:21:46.027762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.027738    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm" event={"ID":"9be5f1ed-bc46-45cb-a778-bbb6e57db950","Type":"ContainerDied","Data":"566c99d089d40d6686cfc4fb606464ba23d6c7854e559fd952cec017b16c2c06"}
Oct 30 22:21:46.029830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.029799    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg" event={"ID":"bf94f1e0-8df4-409d-adcb-7d0423161a7f","Type":"ContainerStarted","Data":"bf11b0edcdf44febfa2162b6c5840a846fcc3da2f49e5a0a6369a3ccf6f903ca"}
Oct 30 22:21:46.030060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.029844    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg" event={"ID":"bf94f1e0-8df4-409d-adcb-7d0423161a7f","Type":"ContainerStarted","Data":"5406eb77d741dea24ddcf9f0862fa5b318fae3c413fc7c1aaa430dadb2859b93"}
Oct 30 22:21:46.103232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.103136    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg" podStartSLOduration=2.10311488 podStartE2EDuration="2.10311488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:46.096070685 +0000 UTC m=+890.135009582" watchObservedRunningTime="2026-10-30 22:21:46.10311488 +0000 UTC m=+890.142053767"
Oct 30 22:21:46.241354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.241316    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:21:46.241542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.241383    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:21:46.936168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.936130    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:46.937312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.937202    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:46.938838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:46.938812    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:21:47.062760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.062715    4581 generic.go:356] "Generic (PLEG): container finished" podID="bf94f1e0-8df4-409d-adcb-7d0423161a7f" containerID="bf11b0edcdf44febfa2162b6c5840a846fcc3da2f49e5a0a6369a3ccf6f903ca" exitCode=0
Oct 30 22:21:47.063460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.063432    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg" event={"ID":"bf94f1e0-8df4-409d-adcb-7d0423161a7f","Type":"ContainerDied","Data":"bf11b0edcdf44febfa2162b6c5840a846fcc3da2f49e5a0a6369a3ccf6f903ca"}
Oct 30 22:21:47.272662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.272625    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:47.274814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.274394    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:47.275458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.275437    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:21:47.330584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.330553    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:47.388685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.387899    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-openshift-ca\") pod \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") "
Oct 30 22:21:47.388685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.388197    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be5f1ed-bc46-45cb-a778-bbb6e57db950-kube-api-access-s8mdd\" (UniqueName: \"kubernetes.io/projected/9be5f1ed-bc46-45cb-a778-bbb6e57db950-kube-api-access-s8mdd\") pod \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") "
Oct 30 22:21:47.388685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.388288    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-xks-ca\") pod \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\" (UID: \"9be5f1ed-bc46-45cb-a778-bbb6e57db950\") "
Oct 30 22:21:47.390320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.390004    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-openshift-ca" pod "9be5f1ed-bc46-45cb-a778-bbb6e57db950" (UID: "9be5f1ed-bc46-45cb-a778-bbb6e57db950"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:47.395734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.395699    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-xks-ca" pod "9be5f1ed-bc46-45cb-a778-bbb6e57db950" (UID: "9be5f1ed-bc46-45cb-a778-bbb6e57db950"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:47.398880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.398829    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be5f1ed-bc46-45cb-a778-bbb6e57db950-kube-api-access-s8mdd" pod "9be5f1ed-bc46-45cb-a778-bbb6e57db950" (UID: "9be5f1ed-bc46-45cb-a778-bbb6e57db950"). InnerVolumeSpecName "kube-api-access-s8mdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:47.491156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.491067    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:47.491156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.491096    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8mdd\" (UniqueName: \"kubernetes.io/projected/9be5f1ed-bc46-45cb-a778-bbb6e57db950-kube-api-access-s8mdd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:47.491156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.491107    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be5f1ed-bc46-45cb-a778-bbb6e57db950-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:47.660201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.660162    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"]
Oct 30 22:21:47.660800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.660786    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9be5f1ed-bc46-45cb-a778-bbb6e57db950" containerName="revoke-keys"
Oct 30 22:21:47.660863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.660801    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be5f1ed-bc46-45cb-a778-bbb6e57db950" containerName="revoke-keys"
Oct 30 22:21:47.660954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.660945    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be5f1ed-bc46-45cb-a778-bbb6e57db950" containerName="revoke-keys"
Oct 30 22:21:47.664814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.664793    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.683651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.683621    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"]
Oct 30 22:21:47.796186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.796093    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wr9\" (UniqueName: \"kubernetes.io/projected/27af7364-b7f5-407f-8b92-d6a55ba6a092-kube-api-access-76wr9\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.796369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.796289    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.796369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.796347    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.897958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.897925    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.898135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.898033    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.898308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.898284    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wr9\" (UniqueName: \"kubernetes.io/projected/27af7364-b7f5-407f-8b92-d6a55ba6a092-kube-api-access-76wr9\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.898882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.898810    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.900960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.900924    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.916104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.916073    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wr9\" (UniqueName: \"kubernetes.io/projected/27af7364-b7f5-407f-8b92-d6a55ba6a092-kube-api-access-76wr9\") pod \"maas-api-revoke-sub-437503af-9v7cl\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:47.981910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:47.981869    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:48.083314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.083206    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm" event={"ID":"9be5f1ed-bc46-45cb-a778-bbb6e57db950","Type":"ContainerDied","Data":"1ba48d77d314ee5c43e106b6cafb5e683fe5f9526981f86cfda71317a6d22ad3"}
Oct 30 22:21:48.083495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.083347    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ba48d77d314ee5c43e106b6cafb5e683fe5f9526981f86cfda71317a6d22ad3"
Oct 30 22:21:48.083902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.083879    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"
Oct 30 22:21:48.278881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.278846    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:48.301866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.301841    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"]
Oct 30 22:21:48.303666 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:21:48.303603    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27af7364_b7f5_407f_8b92_d6a55ba6a092.slice/crio-8f8bd9eed6bdb03ddfe0f930839185993026505f00f8754d74938eaf099dfc0e WatchSource:0}: Error finding container 8f8bd9eed6bdb03ddfe0f930839185993026505f00f8754d74938eaf099dfc0e: Status 404 returned error can't find the container with id 8f8bd9eed6bdb03ddfe0f930839185993026505f00f8754d74938eaf099dfc0e
Oct 30 22:21:48.409231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.409181    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf94f1e0-8df4-409d-adcb-7d0423161a7f-kube-api-access-2k4zx\" (UniqueName: \"kubernetes.io/projected/bf94f1e0-8df4-409d-adcb-7d0423161a7f-kube-api-access-2k4zx\") pod \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") "
Oct 30 22:21:48.409420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.409318    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-openshift-ca\") pod \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") "
Oct 30 22:21:48.409505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.409490    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-xks-ca\") pod \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\" (UID: \"bf94f1e0-8df4-409d-adcb-7d0423161a7f\") "
Oct 30 22:21:48.409772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.409734    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-openshift-ca" pod "bf94f1e0-8df4-409d-adcb-7d0423161a7f" (UID: "bf94f1e0-8df4-409d-adcb-7d0423161a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:48.410188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.410139    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:48.411623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.411598    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf94f1e0-8df4-409d-adcb-7d0423161a7f-kube-api-access-2k4zx" pod "bf94f1e0-8df4-409d-adcb-7d0423161a7f" (UID: "bf94f1e0-8df4-409d-adcb-7d0423161a7f"). InnerVolumeSpecName "kube-api-access-2k4zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:48.411750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.411728    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-xks-ca" pod "bf94f1e0-8df4-409d-adcb-7d0423161a7f" (UID: "bf94f1e0-8df4-409d-adcb-7d0423161a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:48.511923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.511880    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94f1e0-8df4-409d-adcb-7d0423161a7f-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:48.511923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:48.511913    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k4zx\" (UniqueName: \"kubernetes.io/projected/bf94f1e0-8df4-409d-adcb-7d0423161a7f-kube-api-access-2k4zx\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:49.095929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:49.095890    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl" event={"ID":"27af7364-b7f5-407f-8b92-d6a55ba6a092","Type":"ContainerStarted","Data":"95a641b67198021cc58b6c73842f76fb393159369d1729646350e61f2cf9ae00"}
Oct 30 22:21:49.096422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:49.095954    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl" event={"ID":"27af7364-b7f5-407f-8b92-d6a55ba6a092","Type":"ContainerStarted","Data":"8f8bd9eed6bdb03ddfe0f930839185993026505f00f8754d74938eaf099dfc0e"}
Oct 30 22:21:49.097810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:49.097783    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"
Oct 30 22:21:49.097963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:49.097810    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg" event={"ID":"bf94f1e0-8df4-409d-adcb-7d0423161a7f","Type":"ContainerDied","Data":"5406eb77d741dea24ddcf9f0862fa5b318fae3c413fc7c1aaa430dadb2859b93"}
Oct 30 22:21:49.097963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:49.097851    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5406eb77d741dea24ddcf9f0862fa5b318fae3c413fc7c1aaa430dadb2859b93"
Oct 30 22:21:49.139610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:49.139554    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl" podStartSLOduration=2.139524306 podStartE2EDuration="2.139524306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:49.12912986 +0000 UTC m=+893.168068747" watchObservedRunningTime="2026-10-30 22:21:49.139524306 +0000 UTC m=+893.178463202"
Oct 30 22:21:50.107766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:50.107726    4581 generic.go:356] "Generic (PLEG): container finished" podID="27af7364-b7f5-407f-8b92-d6a55ba6a092" containerID="95a641b67198021cc58b6c73842f76fb393159369d1729646350e61f2cf9ae00" exitCode=0
Oct 30 22:21:50.108171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:50.107810    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl" event={"ID":"27af7364-b7f5-407f-8b92-d6a55ba6a092","Type":"ContainerDied","Data":"95a641b67198021cc58b6c73842f76fb393159369d1729646350e61f2cf9ae00"}
Oct 30 22:21:51.322452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.322425    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:51.454316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.454204    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-xks-ca\") pod \"27af7364-b7f5-407f-8b92-d6a55ba6a092\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") "
Oct 30 22:21:51.454316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.454296    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-openshift-ca\") pod \"27af7364-b7f5-407f-8b92-d6a55ba6a092\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") "
Oct 30 22:21:51.454638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.454354    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27af7364-b7f5-407f-8b92-d6a55ba6a092-kube-api-access-76wr9\" (UniqueName: \"kubernetes.io/projected/27af7364-b7f5-407f-8b92-d6a55ba6a092-kube-api-access-76wr9\") pod \"27af7364-b7f5-407f-8b92-d6a55ba6a092\" (UID: \"27af7364-b7f5-407f-8b92-d6a55ba6a092\") "
Oct 30 22:21:51.454703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.454682    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-openshift-ca" pod "27af7364-b7f5-407f-8b92-d6a55ba6a092" (UID: "27af7364-b7f5-407f-8b92-d6a55ba6a092"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:51.455371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.455344    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:51.456858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.456828    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-xks-ca" pod "27af7364-b7f5-407f-8b92-d6a55ba6a092" (UID: "27af7364-b7f5-407f-8b92-d6a55ba6a092"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:51.457390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.457363    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27af7364-b7f5-407f-8b92-d6a55ba6a092-kube-api-access-76wr9" pod "27af7364-b7f5-407f-8b92-d6a55ba6a092" (UID: "27af7364-b7f5-407f-8b92-d6a55ba6a092"). InnerVolumeSpecName "kube-api-access-76wr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:51.556817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.556780    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27af7364-b7f5-407f-8b92-d6a55ba6a092-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:51.556817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:51.556814    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wr9\" (UniqueName: \"kubernetes.io/projected/27af7364-b7f5-407f-8b92-d6a55ba6a092-kube-api-access-76wr9\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:52.127151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.127126    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"
Oct 30 22:21:52.127151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.127147    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl" event={"ID":"27af7364-b7f5-407f-8b92-d6a55ba6a092","Type":"ContainerDied","Data":"8f8bd9eed6bdb03ddfe0f930839185993026505f00f8754d74938eaf099dfc0e"}
Oct 30 22:21:52.127422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.127191    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f8bd9eed6bdb03ddfe0f930839185993026505f00f8754d74938eaf099dfc0e"
Oct 30 22:21:52.549903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.549776    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"]
Oct 30 22:21:52.551462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.551431    4581 cpu_manager.go:414] "containerMap: removing container" podUID="27af7364-b7f5-407f-8b92-d6a55ba6a092" containerName="revoke-keys"
Oct 30 22:21:52.551462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.551457    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27af7364-b7f5-407f-8b92-d6a55ba6a092" containerName="revoke-keys"
Oct 30 22:21:52.551622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.551500    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bf94f1e0-8df4-409d-adcb-7d0423161a7f" containerName="revoke-keys"
Oct 30 22:21:52.551622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.551521    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf94f1e0-8df4-409d-adcb-7d0423161a7f" containerName="revoke-keys"
Oct 30 22:21:52.551686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.551674    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="27af7364-b7f5-407f-8b92-d6a55ba6a092" containerName="revoke-keys"
Oct 30 22:21:52.551719 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.551689    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf94f1e0-8df4-409d-adcb-7d0423161a7f" containerName="revoke-keys"
Oct 30 22:21:52.556235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.556198    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.611087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.611059    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"]
Oct 30 22:21:52.674063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674021    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b191633-9fa0-4662-9221-95b2540c3efc-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.674296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674201    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n8jk\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-kube-api-access-5n8jk\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.674369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674325    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.674425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674376    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.674425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674399    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.674539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674500    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.674595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674545    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.674595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674580    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.674716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.674698    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b191633-9fa0-4662-9221-95b2540c3efc-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.775828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.775784    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776092    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776137    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776201    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776270    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776321    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776375    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b191633-9fa0-4662-9221-95b2540c3efc-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776454    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b191633-9fa0-4662-9221-95b2540c3efc-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776550    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776583    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n8jk\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-kube-api-access-5n8jk\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776606    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776698    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.776985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.776948    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.777096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.777054    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b191633-9fa0-4662-9221-95b2540c3efc-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.779185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.779157    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.779360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.779338    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b191633-9fa0-4662-9221-95b2540c3efc-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.797405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.797369    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n8jk\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-kube-api-access-5n8jk\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.797547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.797460    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.877983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.877940    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:52.960635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.958845    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"]
Oct 30 22:21:52.970417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.970335    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:52.998337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:52.997763    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"]
Oct 30 22:21:53.083282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.083248    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.083472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.083318    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.083472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.083439    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.083654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.083626    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.083734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.083702    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.083799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.083731    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.083951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.083918    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.084000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.083965    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.084037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.084002    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9rj\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-kube-api-access-rl9rj\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.185975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.185884    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.185975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.185941    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.185975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.185958    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186342    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186385    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186422    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9rj\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-kube-api-access-rl9rj\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186468    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186544    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186628    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186706    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186785    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.186874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.186805    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.187163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.187136    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.187360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.187338    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.189856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.189837    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.190290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.190270    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.204043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.204020    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.205816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.205797    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9rj\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-kube-api-access-rl9rj\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.211725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.211703    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"]
Oct 30 22:21:53.212748 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:21:53.212728    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b191633_9fa0_4662_9221_95b2540c3efc.slice/crio-6bf3dcfa5f731a02e0fd560652214abbbc5480b80b430f599ae8f75a10cf055e WatchSource:0}: Error finding container 6bf3dcfa5f731a02e0fd560652214abbbc5480b80b430f599ae8f75a10cf055e: Status 404 returned error can't find the container with id 6bf3dcfa5f731a02e0fd560652214abbbc5480b80b430f599ae8f75a10cf055e
Oct 30 22:21:53.215877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.215838    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:53.216207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.215922    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:53.216207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.215964    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:53.297263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.297189    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:53.646918 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.646891    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"]
Oct 30 22:21:53.647396 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:21:53.646891    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9b24a3_c79b_4d90_80b3_c3c6be2e94cf.slice/crio-cf91f224173cf5f62fefc5d16ec19f1315bbab7ff292527f116b72d7ce4cc94c WatchSource:0}: Error finding container cf91f224173cf5f62fefc5d16ec19f1315bbab7ff292527f116b72d7ce4cc94c: Status 404 returned error can't find the container with id cf91f224173cf5f62fefc5d16ec19f1315bbab7ff292527f116b72d7ce4cc94c
Oct 30 22:21:53.649856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.649820    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:53.649957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.649889    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:53.649957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:53.649921    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:21:54.152702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.152663    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" event={"ID":"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf","Type":"ContainerStarted","Data":"99b0234912798ff5d5e761f2b8192181ed27f73519263d4f4f4ef6bbaf65ecdb"}
Oct 30 22:21:54.153004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.152983    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" event={"ID":"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf","Type":"ContainerStarted","Data":"cf91f224173cf5f62fefc5d16ec19f1315bbab7ff292527f116b72d7ce4cc94c"}
Oct 30 22:21:54.157291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.157257    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" event={"ID":"5b191633-9fa0-4662-9221-95b2540c3efc","Type":"ContainerStarted","Data":"7bad5c7ad95b81d1cd7f48b386fa22497d650e9dfa10654b0a021399cc4a86aa"}
Oct 30 22:21:54.157540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.157522    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" event={"ID":"5b191633-9fa0-4662-9221-95b2540c3efc","Type":"ContainerStarted","Data":"6bf3dcfa5f731a02e0fd560652214abbbc5480b80b430f599ae8f75a10cf055e"}
Oct 30 22:21:54.211602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.211493    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podStartSLOduration=2.211475245 podStartE2EDuration="2.211475245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:54.203260748 +0000 UTC m=+898.242199634" watchObservedRunningTime="2026-10-30 22:21:54.211475245 +0000 UTC m=+898.250414132"
Oct 30 22:21:54.243929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.243869    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podStartSLOduration=2.24385156 podStartE2EDuration="2.24385156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:54.239809976 +0000 UTC m=+898.278748862" watchObservedRunningTime="2026-10-30 22:21:54.24385156 +0000 UTC m=+898.282790446"
Oct 30 22:21:54.323401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.308192    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:21:54.323981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.320624    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:21:54.324167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.324141    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:21:54.339518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.339484    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"]
Oct 30 22:21:54.346294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.346261    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.349081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.349043    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"]
Oct 30 22:21:54.397247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.373652    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:54.397247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.378710    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:54.397247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.378780    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:54.397247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.394487    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:21:54.529554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.529388    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.529554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.529452    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.529825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.529722    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bq7k\" (UniqueName: \"kubernetes.io/projected/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-kube-api-access-4bq7k\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.632816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.632774    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bq7k\" (UniqueName: \"kubernetes.io/projected/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-kube-api-access-4bq7k\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.633029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.633008    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.633107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.633059    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.633854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.633826    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.636453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.636426    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.674303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.654899    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bq7k\" (UniqueName: \"kubernetes.io/projected/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-kube-api-access-4bq7k\") pod \"maas-api-revoke-sub-822617f2-dpwmq\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.744821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.744759    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:54.893037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.891572    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:21:54.895128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.894776    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:21:54.895128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:54.894850    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:21:55.227512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:55.227010    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"]
Oct 30 22:21:55.316241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:55.313507    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:21:55.316241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:55.313574    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:21:55.887244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:55.879520    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:21:55.887244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:55.879585    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:21:56.240408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.240290    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq" event={"ID":"00c1f420-d3ae-4f67-ac70-e8bbdf282a42","Type":"ContainerStarted","Data":"442426bad658a261439e02d7c6b247ebfaa469e5891ce0ca21722941d30182de"}
Oct 30 22:21:56.240408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.240353    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq" event={"ID":"00c1f420-d3ae-4f67-ac70-e8bbdf282a42","Type":"ContainerStarted","Data":"e6a076117e7981343d47bc33169b9fac43358cfd25189f86c554b4ae3ad7f52b"}
Oct 30 22:21:56.317325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.316838    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:21:56.317325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.316897    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:21:56.326563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.321529    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq" podStartSLOduration=2.321508954 podStartE2EDuration="2.321508954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:56.317530841 +0000 UTC m=+900.356469742" watchObservedRunningTime="2026-10-30 22:21:56.321508954 +0000 UTC m=+900.360447841"
Oct 30 22:21:56.726374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.726345    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:21:56.732357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.732332    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:21:56.888731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.888690    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:21:56.889370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:56.889336    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:21:57.249684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:57.249647    4581 generic.go:356] "Generic (PLEG): container finished" podID="00c1f420-d3ae-4f67-ac70-e8bbdf282a42" containerID="442426bad658a261439e02d7c6b247ebfaa469e5891ce0ca21722941d30182de" exitCode=0
Oct 30 22:21:57.249871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:57.249740    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq" event={"ID":"00c1f420-d3ae-4f67-ac70-e8bbdf282a42","Type":"ContainerDied","Data":"442426bad658a261439e02d7c6b247ebfaa469e5891ce0ca21722941d30182de"}
Oct 30 22:21:57.304647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:57.304610    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:21:57.306524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:57.306490    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:21:57.881257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:57.880602    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:21:57.881257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:57.880666    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:21:58.305439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:58.305401    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:21:58.306021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:58.305985    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:21:58.824121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:58.824091    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:58.922913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:58.922426    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:21:58.922913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:58.922491    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:21:58.994815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:58.994715    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-kube-api-access-4bq7k\" (UniqueName: \"kubernetes.io/projected/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-kube-api-access-4bq7k\") pod \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") "
Oct 30 22:21:58.995616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:58.995591    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-xks-ca\") pod \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") "
Oct 30 22:21:59.006612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.006584    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-openshift-ca\") pod \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\" (UID: \"00c1f420-d3ae-4f67-ac70-e8bbdf282a42\") "
Oct 30 22:21:59.007988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.007107    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-openshift-ca" pod "00c1f420-d3ae-4f67-ac70-e8bbdf282a42" (UID: "00c1f420-d3ae-4f67-ac70-e8bbdf282a42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:59.019483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.019312    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-kube-api-access-4bq7k" pod "00c1f420-d3ae-4f67-ac70-e8bbdf282a42" (UID: "00c1f420-d3ae-4f67-ac70-e8bbdf282a42"). InnerVolumeSpecName "kube-api-access-4bq7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:59.024061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.024026    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-xks-ca" pod "00c1f420-d3ae-4f67-ac70-e8bbdf282a42" (UID: "00c1f420-d3ae-4f67-ac70-e8bbdf282a42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:59.111243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.110578    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:59.111243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.110619    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bq7k\" (UniqueName: \"kubernetes.io/projected/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-kube-api-access-4bq7k\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:59.111243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.110637    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c1f420-d3ae-4f67-ac70-e8bbdf282a42-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:21:59.320546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.320028    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:21:59.320546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.320109    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:21:59.326720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.326597    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq" event={"ID":"00c1f420-d3ae-4f67-ac70-e8bbdf282a42","Type":"ContainerDied","Data":"e6a076117e7981343d47bc33169b9fac43358cfd25189f86c554b4ae3ad7f52b"}
Oct 30 22:21:59.326720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.326648    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a076117e7981343d47bc33169b9fac43358cfd25189f86c554b4ae3ad7f52b"
Oct 30 22:21:59.326720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.326708    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"
Oct 30 22:21:59.886244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.885619    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:21:59.886244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:21:59.885685    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:22:00.299299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:00.298891    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:22:00.299299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:00.298939    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:22:00.894242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:00.883719    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:00.894242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:00.883819    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:22:01.298303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:01.297966    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:22:01.298303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:01.298030    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:22:02.003455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.003382    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.063916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.036549    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.246244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.245391    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:22:02.283796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.281366    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.321079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.321027    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.427245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.395575    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:22:02.433242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.430324    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.497244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.493429    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.545341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.508492    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.601245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.600468    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.625247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.615526    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:02.924392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.924346    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:02.924687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:02.924656    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.009373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.009325    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.010698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.010672    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:22:03.232309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.231319    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:22:03.257371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.257318    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.258018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.257995    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:22:03.318248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.297399    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:03.318248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.310871    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:22:03.327423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.327362    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.328277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.328195    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:22:03.352236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.352078    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:03.352236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.352151    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.373250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.353321    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:03.373250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.353378    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.514768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.488203    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:22:03.514768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.488677    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:22:03.514768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.489889    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.515231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.515170    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:22:03.515839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.515819    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:22:03.516042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.516015    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:22:03.574241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.570314    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.574241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.570789    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:22:03.635412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.635359    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.645248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.636084    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:22:03.652433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.652385    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.958693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.957345    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:03.958693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:03.957421    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.026245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.007392    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" podUID="b7e5b659-2162-4552-919d-40c9866fd577" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.33:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.407408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.382690    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.420885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.420855    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:22:04.513072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.512934    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.559895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.453801    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.799874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.799760    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.902093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.600900    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" podUID="b7e5b659-2162-4552-919d-40c9866fd577" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.33:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.902469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.600979    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.912894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.601308    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:05.166313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.601352    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.166313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.601787    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:05.166313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:05.142902    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.166313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.713332    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.166313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.713750    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:22:05.166313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.790148    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:22:05.166313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.593073    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.166313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:04.827367    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": context deadline exceeded"
Oct 30 22:22:05.728312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:05.519365    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.728312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:05.520576    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4" podUID="17847819-d7b1-4946-8a3c-326d50cdd08e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.728312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:05.622479    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.804544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:05.804481    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.955439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:05.955381    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:22:05.956131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:05.956095    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.134302 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:22:06.102340    4581 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.47s"
Oct 30 22:22:06.214942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.170344    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.235842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.170383    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.236108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.170407    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.236325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.170538    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.236481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.170562    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.261669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.170579    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.261967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.171322    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:06.262108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.262082    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.528186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.528082    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.555546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.555512    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:22:06.653243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.631375    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.752383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.740010    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:22:06.752383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.740123    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:22:06.752383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.744752    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:22:06.860151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.859320    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.861747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.861630    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:06.861747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.861689    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.974411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.956298    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.974411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:06.956707    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.580244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.576253    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.580244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.576731    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.580244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.576992    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.605655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.605609    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 22:22:07.605961 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.605933    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.606393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.606369    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.613245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.609497    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.627006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.614718    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.658416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.658381    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:07.658684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.658657    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.673389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.654570    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.708275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.687413    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.889393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:07.876859    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.013959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.013915    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.032389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.024324    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:22:08.599688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.599641    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.600208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.599898    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:22:08.600208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.600069    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:08.616262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.614093    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.616262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.614431    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:08.616262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.614628    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:08.642603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.642559    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:22:08.718603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.718561    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:22:08.841679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.840486    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:08.841989 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.841956    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.927205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.927154    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:22:08.927572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.927553    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:22:08.927712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.927697    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"]
Oct 30 22:22:08.928848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.928824    4581 cpu_manager.go:414] "containerMap: removing container" podUID="00c1f420-d3ae-4f67-ac70-e8bbdf282a42" containerName="revoke-keys"
Oct 30 22:22:08.928966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.928954    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c1f420-d3ae-4f67-ac70-e8bbdf282a42" containerName="revoke-keys"
Oct 30 22:22:08.929259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:08.929245    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c1f420-d3ae-4f67-ac70-e8bbdf282a42" containerName="revoke-keys"
Oct 30 22:22:09.050448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.050416    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"]
Oct 30 22:22:09.050777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.050762    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:22:09.055201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.055160    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:09.055478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.055448    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:22:09.061542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.061519    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:22:09.062104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.062085    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:22:09.062886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.058393    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.063918 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.063899    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:22:09.064655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.064640    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:22:09.070256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.060098    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:22:09.080244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.072773    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:09.080244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.079162    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:09.080244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.079281    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:09.085555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.083648    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:09.087498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.087475    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:22:09.088511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.088491    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:22:09.112241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.098037    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:22:09.170592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.170543    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:22:09.176609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.176582    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:22:09.180651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.180535    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:22:09.246208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.246164    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed77a059-e350-4394-9e32-0901e740b159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.250090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.250059    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed77a059-e350-4394-9e32-0901e740b159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.250814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.250782    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwkc6\" (UniqueName: \"kubernetes.io/projected/ed77a059-e350-4394-9e32-0901e740b159-kube-api-access-jwkc6\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.353012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.352962    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed77a059-e350-4394-9e32-0901e740b159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.354336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.353229    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwkc6\" (UniqueName: \"kubernetes.io/projected/ed77a059-e350-4394-9e32-0901e740b159-kube-api-access-jwkc6\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.354336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.353378    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed77a059-e350-4394-9e32-0901e740b159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.354336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.354016    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed77a059-e350-4394-9e32-0901e740b159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.363878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.363842    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:22:09.364724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.364691    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed77a059-e350-4394-9e32-0901e740b159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.371995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.371960    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwkc6\" (UniqueName: \"kubernetes.io/projected/ed77a059-e350-4394-9e32-0901e740b159-kube-api-access-jwkc6\") pod \"maas-api-revoke-sub-aa65e8a4-dzlvf\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.481309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.481186    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:09.841723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:09.841692    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"]
Oct 30 22:22:09.843454 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:09.843424    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded77a059_e350_4394_9e32_0901e740b159.slice/crio-2beca8534758a38f60caa111558118d71f13783877f26839120d0857ec7c42fe WatchSource:0}: Error finding container 2beca8534758a38f60caa111558118d71f13783877f26839120d0857ec7c42fe: Status 404 returned error can't find the container with id 2beca8534758a38f60caa111558118d71f13783877f26839120d0857ec7c42fe
Oct 30 22:22:10.374654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.374621    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"]
Oct 30 22:22:10.379718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.379654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf" event={"ID":"ed77a059-e350-4394-9e32-0901e740b159","Type":"ContainerStarted","Data":"e9ef0f43c91170e5a025622da72ebabde4c72220a26a035559ba21b2d3cb38f3"}
Oct 30 22:22:10.379933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.379729    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.379933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.379754    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf" event={"ID":"ed77a059-e350-4394-9e32-0901e740b159","Type":"ContainerStarted","Data":"2beca8534758a38f60caa111558118d71f13783877f26839120d0857ec7c42fe"}
Oct 30 22:22:10.403723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.403112    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5l7l\" (UniqueName: \"kubernetes.io/projected/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-kube-api-access-h5l7l\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.403723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.403302    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.403723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.403414    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.417196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.417153    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"]
Oct 30 22:22:10.424609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.424553    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf" podStartSLOduration=4.424535486 podStartE2EDuration="4.424535486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:10.413472868 +0000 UTC m=+914.452411754" watchObservedRunningTime="2026-10-30 22:22:10.424535486 +0000 UTC m=+914.463474430"
Oct 30 22:22:10.505685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.505651    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.505848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.505796    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5l7l\" (UniqueName: \"kubernetes.io/projected/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-kube-api-access-h5l7l\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.505931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.505863    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.506556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.506531    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.508346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.508326    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.525400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.525328    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5l7l\" (UniqueName: \"kubernetes.io/projected/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-kube-api-access-h5l7l\") pod \"maas-api-revoke-sub-4bb8d901-q5mdr\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.722889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.722853    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:10.760570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.760538    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"]
Oct 30 22:22:10.766401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.766377    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:10.807002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.806947    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"]
Oct 30 22:22:10.833180 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:22:10.833138    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:22:10.910922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.910881    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cq84\" (UniqueName: \"kubernetes.io/projected/aadb9d28-ad61-4523-b3b2-9bfb95125858-kube-api-access-5cq84\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:10.911457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.910943    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:10.911457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:10.911040    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:11.012478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.012444    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cq84\" (UniqueName: \"kubernetes.io/projected/aadb9d28-ad61-4523-b3b2-9bfb95125858-kube-api-access-5cq84\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:11.012654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.012496    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:11.012654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.012534    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:11.013193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.013161    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:11.016013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.015985    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:11.029861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.029787    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cq84\" (UniqueName: \"kubernetes.io/projected/aadb9d28-ad61-4523-b3b2-9bfb95125858-kube-api-access-5cq84\") pod \"maas-api-revoke-sub-27ca336c-lnt9f\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:11.046980 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:11.046950    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd0f3ad_37c2_42b0_a475_5ef1ada45958.slice/crio-ac265076a5dc50875c4f0b2f324b533f3153bb6cbb73825923ad9514e4c312cc WatchSource:0}: Error finding container ac265076a5dc50875c4f0b2f324b533f3153bb6cbb73825923ad9514e4c312cc: Status 404 returned error can't find the container with id ac265076a5dc50875c4f0b2f324b533f3153bb6cbb73825923ad9514e4c312cc
Oct 30 22:22:11.047240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.047202    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"]
Oct 30 22:22:11.090639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.090599    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:11.386992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.386956    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr" event={"ID":"dcd0f3ad-37c2-42b0-a475-5ef1ada45958","Type":"ContainerStarted","Data":"ac265076a5dc50875c4f0b2f324b533f3153bb6cbb73825923ad9514e4c312cc"}
Oct 30 22:22:11.388903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.388872    4581 generic.go:356] "Generic (PLEG): container finished" podID="ed77a059-e350-4394-9e32-0901e740b159" containerID="e9ef0f43c91170e5a025622da72ebabde4c72220a26a035559ba21b2d3cb38f3" exitCode=0
Oct 30 22:22:11.389069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.388912    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf" event={"ID":"ed77a059-e350-4394-9e32-0901e740b159","Type":"ContainerDied","Data":"e9ef0f43c91170e5a025622da72ebabde4c72220a26a035559ba21b2d3cb38f3"}
Oct 30 22:22:11.410750 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:11.410720    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadb9d28_ad61_4523_b3b2_9bfb95125858.slice/crio-b292e6ec9d7a9070feab95cc39fa1894c4883f746be89541e968e3358af423da WatchSource:0}: Error finding container b292e6ec9d7a9070feab95cc39fa1894c4883f746be89541e968e3358af423da: Status 404 returned error can't find the container with id b292e6ec9d7a9070feab95cc39fa1894c4883f746be89541e968e3358af423da
Oct 30 22:22:11.424253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:11.421463    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"]
Oct 30 22:22:12.399960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.399923    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f" event={"ID":"aadb9d28-ad61-4523-b3b2-9bfb95125858","Type":"ContainerStarted","Data":"9ac8568bac98d94a711a4fc6c418fcd5e1181d92469fe9d8e23a71f7eabe1564"}
Oct 30 22:22:12.400498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.400023    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f" event={"ID":"aadb9d28-ad61-4523-b3b2-9bfb95125858","Type":"ContainerStarted","Data":"b292e6ec9d7a9070feab95cc39fa1894c4883f746be89541e968e3358af423da"}
Oct 30 22:22:12.401945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.401916    4581 generic.go:356] "Generic (PLEG): container finished" podID="dcd0f3ad-37c2-42b0-a475-5ef1ada45958" containerID="fffd2044bdd08286a87bc49f1c26fd8727cf1fb59c5290c785a83f63b5eebc78" exitCode=0
Oct 30 22:22:12.402079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.401997    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr" event={"ID":"dcd0f3ad-37c2-42b0-a475-5ef1ada45958","Type":"ContainerDied","Data":"fffd2044bdd08286a87bc49f1c26fd8727cf1fb59c5290c785a83f63b5eebc78"}
Oct 30 22:22:12.435104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.435047    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f" podStartSLOduration=2.435032026 podStartE2EDuration="2.435032026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:12.434680378 +0000 UTC m=+916.473619265" watchObservedRunningTime="2026-10-30 22:22:12.435032026 +0000 UTC m=+916.473970911"
Oct 30 22:22:12.582879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.582846    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:12.638345 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.638313    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed77a059-e350-4394-9e32-0901e740b159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed77a059-e350-4394-9e32-0901e740b159-maas-api-xks-ca\") pod \"ed77a059-e350-4394-9e32-0901e740b159\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") "
Oct 30 22:22:12.638499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.638409    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed77a059-e350-4394-9e32-0901e740b159-kube-api-access-jwkc6\" (UniqueName: \"kubernetes.io/projected/ed77a059-e350-4394-9e32-0901e740b159-kube-api-access-jwkc6\") pod \"ed77a059-e350-4394-9e32-0901e740b159\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") "
Oct 30 22:22:12.638606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.638590    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed77a059-e350-4394-9e32-0901e740b159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed77a059-e350-4394-9e32-0901e740b159-maas-api-openshift-ca\") pod \"ed77a059-e350-4394-9e32-0901e740b159\" (UID: \"ed77a059-e350-4394-9e32-0901e740b159\") "
Oct 30 22:22:12.639014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.638980    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed77a059-e350-4394-9e32-0901e740b159-maas-api-openshift-ca" pod "ed77a059-e350-4394-9e32-0901e740b159" (UID: "ed77a059-e350-4394-9e32-0901e740b159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:12.639595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.639569    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed77a059-e350-4394-9e32-0901e740b159-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:12.640971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.640942    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed77a059-e350-4394-9e32-0901e740b159-maas-api-xks-ca" pod "ed77a059-e350-4394-9e32-0901e740b159" (UID: "ed77a059-e350-4394-9e32-0901e740b159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:12.641099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.640992    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed77a059-e350-4394-9e32-0901e740b159-kube-api-access-jwkc6" pod "ed77a059-e350-4394-9e32-0901e740b159" (UID: "ed77a059-e350-4394-9e32-0901e740b159"). InnerVolumeSpecName "kube-api-access-jwkc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:12.741101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.741057    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed77a059-e350-4394-9e32-0901e740b159-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:12.741101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:12.741093    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwkc6\" (UniqueName: \"kubernetes.io/projected/ed77a059-e350-4394-9e32-0901e740b159-kube-api-access-jwkc6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:13.413855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.413817    4581 generic.go:356] "Generic (PLEG): container finished" podID="aadb9d28-ad61-4523-b3b2-9bfb95125858" containerID="9ac8568bac98d94a711a4fc6c418fcd5e1181d92469fe9d8e23a71f7eabe1564" exitCode=0
Oct 30 22:22:13.414467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.413895    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f" event={"ID":"aadb9d28-ad61-4523-b3b2-9bfb95125858","Type":"ContainerDied","Data":"9ac8568bac98d94a711a4fc6c418fcd5e1181d92469fe9d8e23a71f7eabe1564"}
Oct 30 22:22:13.416010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.415982    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"
Oct 30 22:22:13.416117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.416010    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf" event={"ID":"ed77a059-e350-4394-9e32-0901e740b159","Type":"ContainerDied","Data":"2beca8534758a38f60caa111558118d71f13783877f26839120d0857ec7c42fe"}
Oct 30 22:22:13.416117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.416056    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2beca8534758a38f60caa111558118d71f13783877f26839120d0857ec7c42fe"
Oct 30 22:22:13.458347 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.458316    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"]
Oct 30 22:22:13.459058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.459042    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ed77a059-e350-4394-9e32-0901e740b159" containerName="revoke-keys"
Oct 30 22:22:13.459114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.459060    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed77a059-e350-4394-9e32-0901e740b159" containerName="revoke-keys"
Oct 30 22:22:13.459254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.459243    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed77a059-e350-4394-9e32-0901e740b159" containerName="revoke-keys"
Oct 30 22:22:13.463247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.463207    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.487533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.487498    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"]
Oct 30 22:22:13.553465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.553425    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.553627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.553594    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.553749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.553718    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz94h\" (UniqueName: \"kubernetes.io/projected/e77bc59c-a389-4d9f-8092-dc0ec4953902-kube-api-access-vz94h\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.609648 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.609613    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:13.654628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.654580    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-kube-api-access-h5l7l\" (UniqueName: \"kubernetes.io/projected/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-kube-api-access-h5l7l\") pod \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") "
Oct 30 22:22:13.654803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.654714    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-xks-ca\") pod \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") "
Oct 30 22:22:13.655060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.655032    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-openshift-ca\") pod \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\" (UID: \"dcd0f3ad-37c2-42b0-a475-5ef1ada45958\") "
Oct 30 22:22:13.655562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.655528    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.655680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.655628    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.655738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.655709    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz94h\" (UniqueName: \"kubernetes.io/projected/e77bc59c-a389-4d9f-8092-dc0ec4953902-kube-api-access-vz94h\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.656985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.656960    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.657298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.657268    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-openshift-ca" pod "dcd0f3ad-37c2-42b0-a475-5ef1ada45958" (UID: "dcd0f3ad-37c2-42b0-a475-5ef1ada45958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:13.659644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.659610    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-kube-api-access-h5l7l" pod "dcd0f3ad-37c2-42b0-a475-5ef1ada45958" (UID: "dcd0f3ad-37c2-42b0-a475-5ef1ada45958"). InnerVolumeSpecName "kube-api-access-h5l7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:13.659886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.659855    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.661321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.661290    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-xks-ca" pod "dcd0f3ad-37c2-42b0-a475-5ef1ada45958" (UID: "dcd0f3ad-37c2-42b0-a475-5ef1ada45958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:13.674466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.674435    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz94h\" (UniqueName: \"kubernetes.io/projected/e77bc59c-a389-4d9f-8092-dc0ec4953902-kube-api-access-vz94h\") pod \"maas-api-revoke-sub-406f4171-5vzxf\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:13.757888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.757845    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:13.757888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.757885    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5l7l\" (UniqueName: \"kubernetes.io/projected/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-kube-api-access-h5l7l\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:13.757888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.757895    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd0f3ad-37c2-42b0-a475-5ef1ada45958-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:13.781804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:13.781322    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:14.101801 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:14.101762    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77bc59c_a389_4d9f_8092_dc0ec4953902.slice/crio-bed570ca924bc3016016a4d438ff6a6f8a6fc8dd09a01ad4883cc060f74bdcd0 WatchSource:0}: Error finding container bed570ca924bc3016016a4d438ff6a6f8a6fc8dd09a01ad4883cc060f74bdcd0: Status 404 returned error can't find the container with id bed570ca924bc3016016a4d438ff6a6f8a6fc8dd09a01ad4883cc060f74bdcd0
Oct 30 22:22:14.106265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.105433    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"]
Oct 30 22:22:14.426031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.425989    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf" event={"ID":"e77bc59c-a389-4d9f-8092-dc0ec4953902","Type":"ContainerStarted","Data":"bed570ca924bc3016016a4d438ff6a6f8a6fc8dd09a01ad4883cc060f74bdcd0"}
Oct 30 22:22:14.427931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.427898    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"
Oct 30 22:22:14.427931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.427897    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr" event={"ID":"dcd0f3ad-37c2-42b0-a475-5ef1ada45958","Type":"ContainerDied","Data":"ac265076a5dc50875c4f0b2f324b533f3153bb6cbb73825923ad9514e4c312cc"}
Oct 30 22:22:14.428132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.427956    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac265076a5dc50875c4f0b2f324b533f3153bb6cbb73825923ad9514e4c312cc"
Oct 30 22:22:14.616279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.616252    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:14.673871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.673772    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-openshift-ca\") pod \"aadb9d28-ad61-4523-b3b2-9bfb95125858\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") "
Oct 30 22:22:14.674036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.673904    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadb9d28-ad61-4523-b3b2-9bfb95125858-kube-api-access-5cq84\" (UniqueName: \"kubernetes.io/projected/aadb9d28-ad61-4523-b3b2-9bfb95125858-kube-api-access-5cq84\") pod \"aadb9d28-ad61-4523-b3b2-9bfb95125858\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") "
Oct 30 22:22:14.674036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.673994    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-xks-ca\") pod \"aadb9d28-ad61-4523-b3b2-9bfb95125858\" (UID: \"aadb9d28-ad61-4523-b3b2-9bfb95125858\") "
Oct 30 22:22:14.675800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.675512    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-openshift-ca" pod "aadb9d28-ad61-4523-b3b2-9bfb95125858" (UID: "aadb9d28-ad61-4523-b3b2-9bfb95125858"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:14.677351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.677322    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-xks-ca" pod "aadb9d28-ad61-4523-b3b2-9bfb95125858" (UID: "aadb9d28-ad61-4523-b3b2-9bfb95125858"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:14.678291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.678252    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadb9d28-ad61-4523-b3b2-9bfb95125858-kube-api-access-5cq84" pod "aadb9d28-ad61-4523-b3b2-9bfb95125858" (UID: "aadb9d28-ad61-4523-b3b2-9bfb95125858"). InnerVolumeSpecName "kube-api-access-5cq84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:14.775823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.775784    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:14.775823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.775817    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cq84\" (UniqueName: \"kubernetes.io/projected/aadb9d28-ad61-4523-b3b2-9bfb95125858-kube-api-access-5cq84\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:14.775823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:14.775829    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadb9d28-ad61-4523-b3b2-9bfb95125858-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:15.437956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:15.437873    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"
Oct 30 22:22:15.437956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:15.437908    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f" event={"ID":"aadb9d28-ad61-4523-b3b2-9bfb95125858","Type":"ContainerDied","Data":"b292e6ec9d7a9070feab95cc39fa1894c4883f746be89541e968e3358af423da"}
Oct 30 22:22:15.438458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:15.437960    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b292e6ec9d7a9070feab95cc39fa1894c4883f746be89541e968e3358af423da"
Oct 30 22:22:15.440178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:15.440152    4581 generic.go:356] "Generic (PLEG): container finished" podID="e77bc59c-a389-4d9f-8092-dc0ec4953902" containerID="dffc4f813135171b775ad597f26c69d8fb6ca58fbec8ca4935e13ddcaaaf7b4b" exitCode=0
Oct 30 22:22:15.440339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:15.440263    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf" event={"ID":"e77bc59c-a389-4d9f-8092-dc0ec4953902","Type":"ContainerDied","Data":"dffc4f813135171b775ad597f26c69d8fb6ca58fbec8ca4935e13ddcaaaf7b4b"}
Oct 30 22:22:16.635941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.635910    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:16.709586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.709489    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-xks-ca\") pod \"e77bc59c-a389-4d9f-8092-dc0ec4953902\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") "
Oct 30 22:22:16.709752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.709600    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-openshift-ca\") pod \"e77bc59c-a389-4d9f-8092-dc0ec4953902\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") "
Oct 30 22:22:16.709826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.709801    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77bc59c-a389-4d9f-8092-dc0ec4953902-kube-api-access-vz94h\" (UniqueName: \"kubernetes.io/projected/e77bc59c-a389-4d9f-8092-dc0ec4953902-kube-api-access-vz94h\") pod \"e77bc59c-a389-4d9f-8092-dc0ec4953902\" (UID: \"e77bc59c-a389-4d9f-8092-dc0ec4953902\") "
Oct 30 22:22:16.710012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.709985    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-openshift-ca" pod "e77bc59c-a389-4d9f-8092-dc0ec4953902" (UID: "e77bc59c-a389-4d9f-8092-dc0ec4953902"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:16.712141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.712078    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:16.713451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.713417    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77bc59c-a389-4d9f-8092-dc0ec4953902-kube-api-access-vz94h" pod "e77bc59c-a389-4d9f-8092-dc0ec4953902" (UID: "e77bc59c-a389-4d9f-8092-dc0ec4953902"). InnerVolumeSpecName "kube-api-access-vz94h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:16.713646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.713622    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-xks-ca" pod "e77bc59c-a389-4d9f-8092-dc0ec4953902" (UID: "e77bc59c-a389-4d9f-8092-dc0ec4953902"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:16.814237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.814195    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz94h\" (UniqueName: \"kubernetes.io/projected/e77bc59c-a389-4d9f-8092-dc0ec4953902-kube-api-access-vz94h\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:16.814415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:16.814265    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77bc59c-a389-4d9f-8092-dc0ec4953902-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:17.461528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:17.461494    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"
Oct 30 22:22:17.461528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:17.461503    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf" event={"ID":"e77bc59c-a389-4d9f-8092-dc0ec4953902","Type":"ContainerDied","Data":"bed570ca924bc3016016a4d438ff6a6f8a6fc8dd09a01ad4883cc060f74bdcd0"}
Oct 30 22:22:17.461777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:17.461574    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed570ca924bc3016016a4d438ff6a6f8a6fc8dd09a01ad4883cc060f74bdcd0"
Oct 30 22:22:23.263251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.260328    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.263890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.263834    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.288243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.283257    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.288243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.283818    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.678406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.629372    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.678406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.643440    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.678406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.643683    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:22:23.799803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.799765    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"]
Oct 30 22:22:23.801134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.801104    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e77bc59c-a389-4d9f-8092-dc0ec4953902" containerName="revoke-keys"
Oct 30 22:22:23.801329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.801313    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77bc59c-a389-4d9f-8092-dc0ec4953902" containerName="revoke-keys"
Oct 30 22:22:23.801485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.801475    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aadb9d28-ad61-4523-b3b2-9bfb95125858" containerName="revoke-keys"
Oct 30 22:22:23.801562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.801553    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadb9d28-ad61-4523-b3b2-9bfb95125858" containerName="revoke-keys"
Oct 30 22:22:23.801656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.801648    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dcd0f3ad-37c2-42b0-a475-5ef1ada45958" containerName="revoke-keys"
Oct 30 22:22:23.801732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.801723    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd0f3ad-37c2-42b0-a475-5ef1ada45958" containerName="revoke-keys"
Oct 30 22:22:23.801990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.801978    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadb9d28-ad61-4523-b3b2-9bfb95125858" containerName="revoke-keys"
Oct 30 22:22:23.802073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.802065    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd0f3ad-37c2-42b0-a475-5ef1ada45958" containerName="revoke-keys"
Oct 30 22:22:23.802147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.802139    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77bc59c-a389-4d9f-8092-dc0ec4953902" containerName="revoke-keys"
Oct 30 22:22:23.835244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.827399    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:23.898244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.897842    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"]
Oct 30 22:22:23.937304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.936883    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:22:23.967247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:23.964335    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.046292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.046250    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.058486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.058438    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.058876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.058851    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c55a-1402-4954-a044-f1bf8d75bfee-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.059057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.059034    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64m2w\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-kube-api-access-64m2w\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.064243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.059204    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.064477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.064450    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.064723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.064703    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.091387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.091344    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.091585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.091425    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.091701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.066400    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.169257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.169191    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.247305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.204089    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.247678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.247641    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c55a-1402-4954-a044-f1bf8d75bfee-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.262242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.258333    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64m2w\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-kube-api-access-64m2w\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.262542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.262515    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.266243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.262709    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.272466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.272433    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.272685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.272662    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.272832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.272813    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.281482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.281447    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.335245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.282115    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.337013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.208844    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.337200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.249585    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c55a-1402-4954-a044-f1bf8d75bfee-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.345250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.331057    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.372245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.368770    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:22:24.376253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.336407    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.401241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.336461    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.401241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.336491    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.401241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.358611    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.401241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.335544    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.401241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.391873    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:22:24.401241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.395523    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:22:24.401994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.401973    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:22:24.448242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.425944    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.450944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.430914    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.471749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.471708    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.625328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.625287    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64m2w\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-kube-api-access-64m2w\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.711381 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.711325    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.736582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.711345    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.767245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.742408    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:24.778307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.711398    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.778620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.711376    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.809247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.751915    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:22:24.809247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.786684    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:22:24.809247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.786732    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:22:25.099334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.940040    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.099334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:24.959305    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:22:25.099334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.005379    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:22:25.099334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.009386    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:22:25.099334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.090301    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:25.099334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.090363    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.100294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.099348    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.116254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.107379    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:22:25.214562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.177348    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.244449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.244420    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:22:25.373818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.373765    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.395520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.395457    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.424589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.397400    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.424994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.397886    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.771301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.771261    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.802283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.783864    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:22:25.802283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.797134    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.955246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:25.945240    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.012581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.012536    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.204246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.202599    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.302506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.302447    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.449245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.414991    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.452022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.451967    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.465969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.465847    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.467087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.467035    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.792406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.789346    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.792406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.789783    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.828982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.828942    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.948031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:26.947979    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:27.022246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:27.021333    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.023242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.020898    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.023242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.021241    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.023242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.021454    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.023242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.021507    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.023242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.021717    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.023242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.021928    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.023242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.022137    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.029548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.016096    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.058921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.031057    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 30 22:22:28.058921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.031140    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.077360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.059855    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.077360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.060146    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.112628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.112577    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.014634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:28.976711    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:29.043245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.018484    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:29.055326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.055265    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:29.195713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.044019    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.196915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.043962    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:29.197288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.055586    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:29.201243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.066092    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 22:22:29.201440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.066135    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.201749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.066199    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.222532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.077368    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.227518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.226636    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.227518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.226999    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.227518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.227126    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.227518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.227250    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:22:29.236264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.236185    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" podUID="b7e5b659-2162-4552-919d-40c9866fd577" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.33:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:29.462596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.314418    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:29.487441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.487359    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:29.753813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.753775    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:22:29.754014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.753843    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:22:29.754014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.753897    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:22:29.754014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.753933    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:22:29.757801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.757766    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:22:29.759608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.759586    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:22:29.774668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.774640    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:22:29.774955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.774925    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:22:29.775108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.775097    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:22:29.777893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.775252    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:22:29.778013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.777996    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:22:29.790001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:29.789812    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"]
Oct 30 22:22:30.146324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.146037    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"]
Oct 30 22:22:30.159361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.159333    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.168150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.167997    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" event={"ID":"e082c55a-1402-4954-a044-f1bf8d75bfee","Type":"ContainerStarted","Data":"d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493"}
Oct 30 22:22:30.168492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.168392    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" event={"ID":"e082c55a-1402-4954-a044-f1bf8d75bfee","Type":"ContainerStarted","Data":"e6abb4f884fde58bbec8f8aff0d3e014013701bdd2eba0fa2668e632dc89afa0"}
Oct 30 22:22:30.168683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.168668    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"]
Oct 30 22:22:30.173745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.173718    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:22:30.174019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.174002    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:30.174577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.174540    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:30.174865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.174807    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:30.263495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.263461    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsff\" (UniqueName: \"kubernetes.io/projected/22838863-375b-409f-a5b6-e230d45a0e2c-kube-api-access-hmsff\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.263682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.263612    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.275459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.270761    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.372415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.372373    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.372599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.372554    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.372745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.372727    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsff\" (UniqueName: \"kubernetes.io/projected/22838863-375b-409f-a5b6-e230d45a0e2c-kube-api-access-hmsff\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.373623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.373592    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.376263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.375613    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.379035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.378982    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podStartSLOduration=7.378966723 podStartE2EDuration="7.378966723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:30.37745635 +0000 UTC m=+934.416395237" watchObservedRunningTime="2026-10-30 22:22:30.378966723 +0000 UTC m=+934.417905610"
Oct 30 22:22:30.394533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.394502    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsff\" (UniqueName: \"kubernetes.io/projected/22838863-375b-409f-a5b6-e230d45a0e2c-kube-api-access-hmsff\") pod \"maas-api-revoke-sub-b9955387-thfj8\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.434270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.434154    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"]
Oct 30 22:22:30.447647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.447612    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.459655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.459507    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"]
Oct 30 22:22:30.493845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.493811    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:30.580240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.577607    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvhq6\" (UniqueName: \"kubernetes.io/projected/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-kube-api-access-jvhq6\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.580240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.577661    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.580240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.577786    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.710334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.690568    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.710334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.690866    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvhq6\" (UniqueName: \"kubernetes.io/projected/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-kube-api-access-jvhq6\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.710334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.690896    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.710334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.691769    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.735016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.723497    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.738437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.736987    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvhq6\" (UniqueName: \"kubernetes.io/projected/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-kube-api-access-jvhq6\") pod \"maas-api-revoke-sub-e74dcc53-lhkht\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.763671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.758738    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:30.763671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.758894    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:22:30.763671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.758957    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 22:22:30.780407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.780020    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:30.833616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.833585    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"]
Oct 30 22:22:30.834004 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:30.833965    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22838863_375b_409f_a5b6_e230d45a0e2c.slice/crio-931ffaa689f6a3b73b73e4d2d47d06bd410c97e47435a58dad7ff11c0585ffd6 WatchSource:0}: Error finding container 931ffaa689f6a3b73b73e4d2d47d06bd410c97e47435a58dad7ff11c0585ffd6: Status 404 returned error can't find the container with id 931ffaa689f6a3b73b73e4d2d47d06bd410c97e47435a58dad7ff11c0585ffd6
Oct 30 22:22:30.849090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.849058    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"]
Oct 30 22:22:30.900328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.899774    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"]
Oct 30 22:22:30.900328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:30.899903    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.002168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.002068    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.002336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.002280    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.002430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.002416    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5sk\" (UniqueName: \"kubernetes.io/projected/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-kube-api-access-8d5sk\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.104672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.104639    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.104814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.104747    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5sk\" (UniqueName: \"kubernetes.io/projected/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-kube-api-access-8d5sk\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.104880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.104866    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.105741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.105717    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.107614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.107589    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.124413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.123561    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5sk\" (UniqueName: \"kubernetes.io/projected/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-kube-api-access-8d5sk\") pod \"maas-api-revoke-sub-b903b2f4-lvhzw\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.192189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.192157    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"]
Oct 30 22:22:31.201696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.201663    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8" event={"ID":"22838863-375b-409f-a5b6-e230d45a0e2c","Type":"ContainerStarted","Data":"931ffaa689f6a3b73b73e4d2d47d06bd410c97e47435a58dad7ff11c0585ffd6"}
Oct 30 22:22:31.281248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.278518    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:31.635415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.635390    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"]
Oct 30 22:22:31.635944 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:31.635920    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1eb44e0_1a6c_485b_8d3f_ac110d2b5472.slice/crio-5ed02df06c6365318d468a0e10f06f90ec14b37fee8a2cc5cb82e8e22748c4cc WatchSource:0}: Error finding container 5ed02df06c6365318d468a0e10f06f90ec14b37fee8a2cc5cb82e8e22748c4cc: Status 404 returned error can't find the container with id 5ed02df06c6365318d468a0e10f06f90ec14b37fee8a2cc5cb82e8e22748c4cc
Oct 30 22:22:31.788270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.782053    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:22:31.788270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:31.782112    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 22:22:32.219306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:32.216394    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw" event={"ID":"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472","Type":"ContainerStarted","Data":"5ed02df06c6365318d468a0e10f06f90ec14b37fee8a2cc5cb82e8e22748c4cc"}
Oct 30 22:22:32.219306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:32.218117    4581 generic.go:356] "Generic (PLEG): container finished" podID="22838863-375b-409f-a5b6-e230d45a0e2c" containerID="205ba62da10c74729b61a3e36cf8982d3f8fcc96866da8a09f6c24afd7cd70bd" exitCode=0
Oct 30 22:22:32.219306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:32.218387    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8" event={"ID":"22838863-375b-409f-a5b6-e230d45a0e2c","Type":"ContainerDied","Data":"205ba62da10c74729b61a3e36cf8982d3f8fcc96866da8a09f6c24afd7cd70bd"}
Oct 30 22:22:32.220118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:32.220091    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht" event={"ID":"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d","Type":"ContainerStarted","Data":"107ec363cbc2f4d970b80876180ffc2469709f1c07d939f6575e06a191c01c01"}
Oct 30 22:22:32.220192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:32.220145    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht" event={"ID":"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d","Type":"ContainerStarted","Data":"649a5cffd9c7c2a1678c2b7c040a34e4e8e6fddacd3fb6a73028eb868f774a99"}
Oct 30 22:22:32.311236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:32.311132    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht" podStartSLOduration=2.3111176589999998 podStartE2EDuration="2.311117659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:32.309687763 +0000 UTC m=+936.348626647" watchObservedRunningTime="2026-10-30 22:22:32.311117659 +0000 UTC m=+936.350056575"
Oct 30 22:22:32.753327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:32.752623    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:22:32.753327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:32.752683    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 22:22:33.233104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.233066    4581 generic.go:356] "Generic (PLEG): container finished" podID="4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d" containerID="107ec363cbc2f4d970b80876180ffc2469709f1c07d939f6575e06a191c01c01" exitCode=0
Oct 30 22:22:33.233603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.233150    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht" event={"ID":"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d","Type":"ContainerDied","Data":"107ec363cbc2f4d970b80876180ffc2469709f1c07d939f6575e06a191c01c01"}
Oct 30 22:22:33.235027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.235004    4581 generic.go:356] "Generic (PLEG): container finished" podID="c1eb44e0-1a6c-485b-8d3f-ac110d2b5472" containerID="1ecfbb903fbdbbc9db9c47d1df0b0a7f73d5fb8a6bb0cfcc857fcd80d7912b7a" exitCode=0
Oct 30 22:22:33.235123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.235053    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw" event={"ID":"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472","Type":"ContainerDied","Data":"1ecfbb903fbdbbc9db9c47d1df0b0a7f73d5fb8a6bb0cfcc857fcd80d7912b7a"}
Oct 30 22:22:33.524523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.524495    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:33.644166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.644126    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-xks-ca\") pod \"22838863-375b-409f-a5b6-e230d45a0e2c\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") "
Oct 30 22:22:33.644400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.644246    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-openshift-ca\") pod \"22838863-375b-409f-a5b6-e230d45a0e2c\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") "
Oct 30 22:22:33.644541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.644459    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22838863-375b-409f-a5b6-e230d45a0e2c-kube-api-access-hmsff\" (UniqueName: \"kubernetes.io/projected/22838863-375b-409f-a5b6-e230d45a0e2c-kube-api-access-hmsff\") pod \"22838863-375b-409f-a5b6-e230d45a0e2c\" (UID: \"22838863-375b-409f-a5b6-e230d45a0e2c\") "
Oct 30 22:22:33.644707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.644678    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-openshift-ca" pod "22838863-375b-409f-a5b6-e230d45a0e2c" (UID: "22838863-375b-409f-a5b6-e230d45a0e2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:33.645330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.645267    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:33.646562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.646533    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-xks-ca" pod "22838863-375b-409f-a5b6-e230d45a0e2c" (UID: "22838863-375b-409f-a5b6-e230d45a0e2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:33.646839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.646814    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22838863-375b-409f-a5b6-e230d45a0e2c-kube-api-access-hmsff" pod "22838863-375b-409f-a5b6-e230d45a0e2c" (UID: "22838863-375b-409f-a5b6-e230d45a0e2c"). InnerVolumeSpecName "kube-api-access-hmsff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:33.747191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.747088    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmsff\" (UniqueName: \"kubernetes.io/projected/22838863-375b-409f-a5b6-e230d45a0e2c-kube-api-access-hmsff\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:33.747191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.747124    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22838863-375b-409f-a5b6-e230d45a0e2c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:33.774242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.757646    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:22:33.774242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:33.757719    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 22:22:34.246133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.246101    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"
Oct 30 22:22:34.246548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.246137    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8" event={"ID":"22838863-375b-409f-a5b6-e230d45a0e2c","Type":"ContainerDied","Data":"931ffaa689f6a3b73b73e4d2d47d06bd410c97e47435a58dad7ff11c0585ffd6"}
Oct 30 22:22:34.246548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.246190    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931ffaa689f6a3b73b73e4d2d47d06bd410c97e47435a58dad7ff11c0585ffd6"
Oct 30 22:22:34.535366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.535335    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:34.617623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.617546    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:34.707024 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.706981    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-openshift-ca\") pod \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") "
Oct 30 22:22:34.707264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.707095    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-xks-ca\") pod \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") "
Oct 30 22:22:34.707264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.707183    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-openshift-ca\") pod \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") "
Oct 30 22:22:34.707413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.707266    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-kube-api-access-jvhq6\" (UniqueName: \"kubernetes.io/projected/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-kube-api-access-jvhq6\") pod \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") "
Oct 30 22:22:34.707413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.707343    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-xks-ca\") pod \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\" (UID: \"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d\") "
Oct 30 22:22:34.709168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.709134    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-openshift-ca" pod "4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d" (UID: "4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:34.709400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.709367    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-openshift-ca" pod "c1eb44e0-1a6c-485b-8d3f-ac110d2b5472" (UID: "c1eb44e0-1a6c-485b-8d3f-ac110d2b5472"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:34.711417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.711326    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-xks-ca" pod "4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d" (UID: "4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:34.711417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.711340    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-kube-api-access-jvhq6" pod "4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d" (UID: "4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d"). InnerVolumeSpecName "kube-api-access-jvhq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:34.712664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.712634    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-xks-ca" pod "c1eb44e0-1a6c-485b-8d3f-ac110d2b5472" (UID: "c1eb44e0-1a6c-485b-8d3f-ac110d2b5472"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:34.767278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.766744    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:22:34.767278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.766811    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 22:22:34.813272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.808628    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-kube-api-access-8d5sk\" (UniqueName: \"kubernetes.io/projected/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-kube-api-access-8d5sk\") pod \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\" (UID: \"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472\") "
Oct 30 22:22:34.813272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.809376    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:34.813272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.809399    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:34.813272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.809416    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:34.813272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.809429    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvhq6\" (UniqueName: \"kubernetes.io/projected/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-kube-api-access-jvhq6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:34.813272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.809443    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:34.816384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.816352    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-kube-api-access-8d5sk" pod "c1eb44e0-1a6c-485b-8d3f-ac110d2b5472" (UID: "c1eb44e0-1a6c-485b-8d3f-ac110d2b5472"). InnerVolumeSpecName "kube-api-access-8d5sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:34.911153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:34.911062    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d5sk\" (UniqueName: \"kubernetes.io/projected/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472-kube-api-access-8d5sk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:35.293368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:35.293273    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"
Oct 30 22:22:35.293368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:35.293287    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht" event={"ID":"4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d","Type":"ContainerDied","Data":"649a5cffd9c7c2a1678c2b7c040a34e4e8e6fddacd3fb6a73028eb868f774a99"}
Oct 30 22:22:35.293368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:35.293336    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="649a5cffd9c7c2a1678c2b7c040a34e4e8e6fddacd3fb6a73028eb868f774a99"
Oct 30 22:22:35.295129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:35.295106    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw" event={"ID":"c1eb44e0-1a6c-485b-8d3f-ac110d2b5472","Type":"ContainerDied","Data":"5ed02df06c6365318d468a0e10f06f90ec14b37fee8a2cc5cb82e8e22748c4cc"}
Oct 30 22:22:35.295283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:35.295143    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ed02df06c6365318d468a0e10f06f90ec14b37fee8a2cc5cb82e8e22748c4cc"
Oct 30 22:22:35.295283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:35.295198    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"
Oct 30 22:22:35.772245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:35.769636    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:22:35.772245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:35.769700    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 22:22:36.752292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:36.752182    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:22:36.752669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:36.752289    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 22:22:37.757001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.756964    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:37.757547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.757528    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:37.758179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.758161    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:22:37.954005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.953969    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"]
Oct 30 22:22:37.954661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954646    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d" containerName="revoke-keys"
Oct 30 22:22:37.954715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954662    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d" containerName="revoke-keys"
Oct 30 22:22:37.954715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954679    4581 cpu_manager.go:414] "containerMap: removing container" podUID="22838863-375b-409f-a5b6-e230d45a0e2c" containerName="revoke-keys"
Oct 30 22:22:37.954715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954684    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22838863-375b-409f-a5b6-e230d45a0e2c" containerName="revoke-keys"
Oct 30 22:22:37.954715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954698    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c1eb44e0-1a6c-485b-8d3f-ac110d2b5472" containerName="revoke-keys"
Oct 30 22:22:37.954715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954704    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1eb44e0-1a6c-485b-8d3f-ac110d2b5472" containerName="revoke-keys"
Oct 30 22:22:37.954860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954823    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d" containerName="revoke-keys"
Oct 30 22:22:37.954860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954832    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="22838863-375b-409f-a5b6-e230d45a0e2c" containerName="revoke-keys"
Oct 30 22:22:37.954860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.954840    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1eb44e0-1a6c-485b-8d3f-ac110d2b5472" containerName="revoke-keys"
Oct 30 22:22:37.962612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.962578    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:37.993657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:37.993619    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"]
Oct 30 22:22:38.089333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.089283    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwcdb\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-kube-api-access-hwcdb\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.089658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.089631    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.089916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.089895    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.090157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.090137    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a2f67-6328-40cb-8866-cc4e17b0584d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.090337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.090319    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.090503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.090487    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.090629 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.090614    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.090804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.090788    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.090960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.090944    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.116262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.116193    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"]
Oct 30 22:22:38.121088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.121068    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.167313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.167278    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"]
Oct 30 22:22:38.192729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.192691    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.192919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.192769    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.192919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.192802    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.192919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.192851    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.193117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.192931    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.193117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.192967    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.193117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193020    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.193117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193073    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwcdb\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-kube-api-access-hwcdb\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.193117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193103    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfsw\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-kube-api-access-bxfsw\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.193412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193142    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.193412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193173    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.193412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193209    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.193412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193252    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.193412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193338    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.193412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193399    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.193781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193442    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.193781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193523    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.193781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.193553    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a2f67-6328-40cb-8866-cc4e17b0584d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.194466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.194439    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a2f67-6328-40cb-8866-cc4e17b0584d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.195840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.195810    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.196153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.196122    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.196364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.196340    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.196552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.196523    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.209029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.208965    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.211476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.211438    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.212662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.212637    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.213330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.213311    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwcdb\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-kube-api-access-hwcdb\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.294793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.294701    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.294958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.294794    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.294958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.294893    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.294976    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfsw\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-kube-api-access-bxfsw\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295014    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295045    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295081    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295165    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295252    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295292    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295431    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295505    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.295645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.295622    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.296077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.296054    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.298148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.298122    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.298260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.298149    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.311682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.311655    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.311814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.311706    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfsw\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-kube-api-access-bxfsw\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.320730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.320703    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:38.437771 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.437704    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:38.556763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.556531    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"]
Oct 30 22:22:38.563924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.563897    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.589277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.585689    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"]
Oct 30 22:22:38.663525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.663472    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"]
Oct 30 22:22:38.663946 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:38.663844    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741a2f67_6328_40cb_8866_cc4e17b0584d.slice/crio-d93f69b84025c054b31467859c340a13f3b5d4a3f67600a0b5a9b4346cee3d64 WatchSource:0}: Error finding container d93f69b84025c054b31467859c340a13f3b5d4a3f67600a0b5a9b4346cee3d64: Status 404 returned error can't find the container with id d93f69b84025c054b31467859c340a13f3b5d4a3f67600a0b5a9b4346cee3d64
Oct 30 22:22:38.666286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.666252    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:38.666364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.666318    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:38.666399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.666361    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:38.702291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702211    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.702396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702349    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.702396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702369    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.702396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702395    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.702531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702416    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz59n\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-kube-api-access-tz59n\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.702598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702569    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.702757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702738    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.702819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702784    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.702891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.702871    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.772952 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.772904    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"]
Oct 30 22:22:38.777402 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:38.777354    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b6695b_e9f4_4fd0_8979_d06d71d6a085.slice/crio-4e1df69a5509ce8a13178bfbfa7555809a6b1206bdd75f4af2141a2033ebba86 WatchSource:0}: Error finding container 4e1df69a5509ce8a13178bfbfa7555809a6b1206bdd75f4af2141a2033ebba86: Status 404 returned error can't find the container with id 4e1df69a5509ce8a13178bfbfa7555809a6b1206bdd75f4af2141a2033ebba86
Oct 30 22:22:38.780417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.780385    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:38.780511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.780461    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:38.780511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.780492    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:38.806062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806034    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806081    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806126    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806163    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz59n\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-kube-api-access-tz59n\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806212    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806370    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806423    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806425    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806509    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806677    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.806793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806681    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.807067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.806946    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.807180 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.807062    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.807271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.807178    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.810507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.810484    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.810706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.810689    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.826412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.826389    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.826716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.826694    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz59n\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-kube-api-access-tz59n\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:38.890474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:38.889132    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:39.243939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.243912    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"]
Oct 30 22:22:39.246741 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:39.246714    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f36daa6_0941_48ff_9e00_105f6eb2eeed.slice/crio-53c8607e1780659e9ed860bb8d8f661248e884998abb400e20c2e61e34a4eab5 WatchSource:0}: Error finding container 53c8607e1780659e9ed860bb8d8f661248e884998abb400e20c2e61e34a4eab5: Status 404 returned error can't find the container with id 53c8607e1780659e9ed860bb8d8f661248e884998abb400e20c2e61e34a4eab5
Oct 30 22:22:39.251044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.250998    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:39.251149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.251094    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:39.251205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.251144    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:22:39.293053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.293017    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"]
Oct 30 22:22:39.314128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.314098    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.318708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.318672    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"]
Oct 30 22:22:39.342967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.342602    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:22:39.342967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.342684    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:39.342967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.342757    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:39.342967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.342787    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:39.379003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.376440    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" event={"ID":"90b6695b-e9f4-4fd0-8979-d06d71d6a085","Type":"ContainerStarted","Data":"8acce8cd0e073153aa418904524a83822fcc64acf9947645e2051ad66000ea95"}
Oct 30 22:22:39.379003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.376520    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" event={"ID":"90b6695b-e9f4-4fd0-8979-d06d71d6a085","Type":"ContainerStarted","Data":"4e1df69a5509ce8a13178bfbfa7555809a6b1206bdd75f4af2141a2033ebba86"}
Oct 30 22:22:39.422366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.416096    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podStartSLOduration=1.416075602 podStartE2EDuration="1.416075602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:39.413191907 +0000 UTC m=+943.452130794" watchObservedRunningTime="2026-10-30 22:22:39.416075602 +0000 UTC m=+943.455014488"
Oct 30 22:22:39.422366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.416192    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" event={"ID":"741a2f67-6328-40cb-8866-cc4e17b0584d","Type":"ContainerStarted","Data":"7c71a65413e7ba1114212a26e7f33c30c8f8bc262be2b9eae706fb828ec29afe"}
Oct 30 22:22:39.422366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.416297    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" event={"ID":"741a2f67-6328-40cb-8866-cc4e17b0584d","Type":"ContainerStarted","Data":"d93f69b84025c054b31467859c340a13f3b5d4a3f67600a0b5a9b4346cee3d64"}
Oct 30 22:22:39.422366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.421198    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5sz\" (UniqueName: \"kubernetes.io/projected/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-kube-api-access-xq5sz\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.422366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.421713    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.422366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.421804    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.422366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.422083    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" event={"ID":"8f36daa6-0941-48ff-9e00-105f6eb2eeed","Type":"ContainerStarted","Data":"53c8607e1780659e9ed860bb8d8f661248e884998abb400e20c2e61e34a4eab5"}
Oct 30 22:22:39.446245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.441158    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:39.446245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.445980    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:22:39.446245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.446061    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:22:39.489109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.488805    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podStartSLOduration=2.488783036 podStartE2EDuration="2.488783036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:39.467758105 +0000 UTC m=+943.506697016" watchObservedRunningTime="2026-10-30 22:22:39.488783036 +0000 UTC m=+943.527721925"
Oct 30 22:22:39.524762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.524666    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.525991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.525964    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.526297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.526277    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.527256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.527234    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5sz\" (UniqueName: \"kubernetes.io/projected/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-kube-api-access-xq5sz\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.535375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.535320    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.591242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.590526    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5sz\" (UniqueName: \"kubernetes.io/projected/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-kube-api-access-xq5sz\") pod \"maas-api-revoke-sub-1b5c5900-xrm5k\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:39.661862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:39.661548    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:40.342078 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.342033    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:40.349545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.346357    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.349545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.346416    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:22:40.405179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.405136    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"]
Oct 30 22:22:40.491425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.469284    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.491425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.469363    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:22:40.612105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.612045    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"]
Oct 30 22:22:40.744244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.742669    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"]
Oct 30 22:22:40.744244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.742790    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:40.747198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.746261    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" event={"ID":"8f36daa6-0941-48ff-9e00-105f6eb2eeed","Type":"ContainerStarted","Data":"6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85"}
Oct 30 22:22:40.751262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.750679    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k" event={"ID":"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3","Type":"ContainerStarted","Data":"6b60d6a1ccf9558cbd7873f4a181eb4e93834c4deb3d5efca00ad7fea9c34c20"}
Oct 30 22:22:40.909304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.909266    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:40.909622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.909602    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.909795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.909771    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:22:40.956246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.939286    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc9vl\" (UniqueName: \"kubernetes.io/projected/91279279-1fb9-49d7-bc82-b1519770631b-kube-api-access-hc9vl\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:40.956246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.939343    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:40.956246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.939706    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:40.982248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:40.960267    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podStartSLOduration=2.960247238 podStartE2EDuration="2.960247238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:40.82943296 +0000 UTC m=+944.868371847" watchObservedRunningTime="2026-10-30 22:22:40.960247238 +0000 UTC m=+944.999186122"
Oct 30 22:22:41.060245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.054488    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:41.060245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.054681    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc9vl\" (UniqueName: \"kubernetes.io/projected/91279279-1fb9-49d7-bc82-b1519770631b-kube-api-access-hc9vl\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:41.060245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.054719    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:41.060245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.055682    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:41.317971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.317887    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:41.369468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.369435    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"]
Oct 30 22:22:41.536808 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:22:41.526581    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:22:41.536808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.526838    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.536808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.526886    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:22:41.538456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.537697    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.538456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.537758    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:22:41.563564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.563532    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.587417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.587331    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"]
Oct 30 22:22:41.610256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.604898    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc9vl\" (UniqueName: \"kubernetes.io/projected/91279279-1fb9-49d7-bc82-b1519770631b-kube-api-access-hc9vl\") pod \"maas-api-revoke-sub-f824c259-q6c7f\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:41.635246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.614839    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjs8r\" (UniqueName: \"kubernetes.io/projected/2fc66158-5167-46a8-8191-a0f3ea99d8a9-kube-api-access-mjs8r\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.635625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.635598    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.635940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.635921    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.684245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.684191    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:41.739411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.739364    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjs8r\" (UniqueName: \"kubernetes.io/projected/2fc66158-5167-46a8-8191-a0f3ea99d8a9-kube-api-access-mjs8r\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.765737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.765703    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.767416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.767388    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.768253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.768203    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.855303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.854350    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:41.934244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.931717    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:41.934244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.931838    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.934244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.931882    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:22:41.941018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:41.940987    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjs8r\" (UniqueName: \"kubernetes.io/projected/2fc66158-5167-46a8-8191-a0f3ea99d8a9-kube-api-access-mjs8r\") pod \"maas-api-revoke-sub-e0281e42-n5kjm\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:42.010258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.008643    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k" event={"ID":"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3","Type":"ContainerStarted","Data":"6e42aa695b0164b9d9e51d94f4b230c5672af6f51cc8ed900b00885c333981ed"}
Oct 30 22:22:42.021362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.020588    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:42.217315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.214672    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:22:42.259336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.257960    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:22:42.259336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.258505    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:42.332779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.331770    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.332779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.331833    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:22:42.443596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.443562    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.444285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.444253    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:22:42.511635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.511585    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:42.895282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.895233    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:43.010491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.904522    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:22:43.238267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.947244    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.238623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.238578    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:22:43.238783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.080996    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:43.241673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:42.986400    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:22:43.340685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.324582    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:43.396348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.396301    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:43.427245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.396867    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k" podStartSLOduration=4.396846687 podStartE2EDuration="4.396846687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:42.105910407 +0000 UTC m=+946.144849294" watchObservedRunningTime="2026-10-30 22:22:43.396846687 +0000 UTC m=+947.435785575"
Oct 30 22:22:43.427245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.424704    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:22:43.442493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.442458    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"]
Oct 30 22:22:43.442889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.442858    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.443070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.443044    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:22:43.530471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.518516    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.531289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.531248    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:22:43.847303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.837620    4581 generic.go:356] "Generic (PLEG): container finished" podID="0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3" containerID="6e42aa695b0164b9d9e51d94f4b230c5672af6f51cc8ed900b00885c333981ed" exitCode=0
Oct 30 22:22:43.847303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:43.847107    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k" event={"ID":"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3","Type":"ContainerDied","Data":"6e42aa695b0164b9d9e51d94f4b230c5672af6f51cc8ed900b00885c333981ed"}
Oct 30 22:22:44.150268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.148715    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp" podUID="90d4ed54-8bcf-442b-82c4-cbae47556237" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:44.215858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.209379    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4" podUID="17847819-d7b1-4946-8a3c-326d50cdd08e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:44.215858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.214681    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.215858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.214726    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:22:44.366340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.296103    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:22:44.366340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.322323    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:22:44.375271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.370189    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.395870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.395821    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:22:44.436261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.436148    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:44.451322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.444344    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:22:44.511140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.464099    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:44.511693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.490963    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.511853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.511826    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:22:44.628276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.609990    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:44.688301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.685766    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:22:44.688301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.685833    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:22:44.735242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.728303    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"]
Oct 30 22:22:44.790245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.781278    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": context deadline exceeded"
Oct 30 22:22:44.935246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.897193    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.935246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:44.919311    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:22:45.030306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.025607    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:45.045284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.044432    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:45.335462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.329816    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:45.335462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.330081    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:22:45.409929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.409890    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:22:45.410184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.410153    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:22:45.414574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.414159    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:22:45.489990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.489955    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:22:45.490805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.490771    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:22:45.539296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.521082    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:22:45.545544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.539089    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:45.545544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.542474    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f" event={"ID":"91279279-1fb9-49d7-bc82-b1519770631b","Type":"ContainerStarted","Data":"41ee9ac5d7a4eebcc0e75b73cdd1f434d498714fabd8efc7ce46c96f2224f409"}
Oct 30 22:22:45.545544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.544012    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:22:45.546615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.546590    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm" event={"ID":"2fc66158-5167-46a8-8191-a0f3ea99d8a9","Type":"ContainerStarted","Data":"da5162ae6d3b6b3bd508de5c66d0c9ce2f2486a0f4b97e8eb1f2a5ae0af6aada"}
Oct 30 22:22:45.607695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.581773    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:22:45.611774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.610552    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:22:45.682783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.682750    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:22:45.742139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.737508    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:22:45.903335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.902740    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:45.903335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:45.902801    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:22:46.032297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.032265    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:46.208343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.208296    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-xks-ca\") pod \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") "
Oct 30 22:22:46.208511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.208405    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-kube-api-access-xq5sz\" (UniqueName: \"kubernetes.io/projected/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-kube-api-access-xq5sz\") pod \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") "
Oct 30 22:22:46.208567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.208511    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-openshift-ca\") pod \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\" (UID: \"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3\") "
Oct 30 22:22:46.209006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.208952    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-openshift-ca" pod "0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3" (UID: "0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:46.209652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.209616    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:46.211730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.211700    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-xks-ca" pod "0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3" (UID: "0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:46.211820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.211753    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-kube-api-access-xq5sz" pod "0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3" (UID: "0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3"). InnerVolumeSpecName "kube-api-access-xq5sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:46.311587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.311546    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:46.311587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.311588    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq5sz\" (UniqueName: \"kubernetes.io/projected/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3-kube-api-access-xq5sz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:46.321733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.321703    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:22:46.321963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.321939    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 22:22:46.438612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.438512    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:22:46.438612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.438577    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 22:22:46.635586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.635515    4581 generic.go:356] "Generic (PLEG): container finished" podID="91279279-1fb9-49d7-bc82-b1519770631b" containerID="487260425dcf18a29326d66e31a4b9a98137408d32b3cdbf661ef981ab2018d0" exitCode=0
Oct 30 22:22:46.639054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.635920    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f" event={"ID":"91279279-1fb9-49d7-bc82-b1519770631b","Type":"ContainerDied","Data":"487260425dcf18a29326d66e31a4b9a98137408d32b3cdbf661ef981ab2018d0"}
Oct 30 22:22:46.640176 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.640144    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k" event={"ID":"0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3","Type":"ContainerDied","Data":"6b60d6a1ccf9558cbd7873f4a181eb4e93834c4deb3d5efca00ad7fea9c34c20"}
Oct 30 22:22:46.640403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.640384    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b60d6a1ccf9558cbd7873f4a181eb4e93834c4deb3d5efca00ad7fea9c34c20"
Oct 30 22:22:46.640593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.640577    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"
Oct 30 22:22:46.646055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.646028    4581 generic.go:356] "Generic (PLEG): container finished" podID="2fc66158-5167-46a8-8191-a0f3ea99d8a9" containerID="74c6b74f59f3671a442c4e33e920c69e459d8c346bf565fed26ca9f94fd3c9d3" exitCode=0
Oct 30 22:22:46.646586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.646550    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm" event={"ID":"2fc66158-5167-46a8-8191-a0f3ea99d8a9","Type":"ContainerDied","Data":"74c6b74f59f3671a442c4e33e920c69e459d8c346bf565fed26ca9f94fd3c9d3"}
Oct 30 22:22:46.890744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.890698    4581 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:46.891326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:46.890758    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:22:47.521243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:47.520705    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:47.522634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:47.522259    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:47.528814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:47.525860    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:22:47.552629 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:47.552596    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:47.553997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:47.553974    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:47.556919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:47.556874    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:22:47.978761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:47.978451    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:48.024596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.024566    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:48.158157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.158106    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-openshift-ca\") pod \"91279279-1fb9-49d7-bc82-b1519770631b\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") "
Oct 30 22:22:48.158379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.158328    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-xks-ca\") pod \"91279279-1fb9-49d7-bc82-b1519770631b\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") "
Oct 30 22:22:48.158491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.158468    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-xks-ca\") pod \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") "
Oct 30 22:22:48.158586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.158542    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-openshift-ca" pod "91279279-1fb9-49d7-bc82-b1519770631b" (UID: "91279279-1fb9-49d7-bc82-b1519770631b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:48.158586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.158545    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc66158-5167-46a8-8191-a0f3ea99d8a9-kube-api-access-mjs8r\" (UniqueName: \"kubernetes.io/projected/2fc66158-5167-46a8-8191-a0f3ea99d8a9-kube-api-access-mjs8r\") pod \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") "
Oct 30 22:22:48.158720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.158651    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91279279-1fb9-49d7-bc82-b1519770631b-kube-api-access-hc9vl\" (UniqueName: \"kubernetes.io/projected/91279279-1fb9-49d7-bc82-b1519770631b-kube-api-access-hc9vl\") pod \"91279279-1fb9-49d7-bc82-b1519770631b\" (UID: \"91279279-1fb9-49d7-bc82-b1519770631b\") "
Oct 30 22:22:48.158774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.158733    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-openshift-ca\") pod \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\" (UID: \"2fc66158-5167-46a8-8191-a0f3ea99d8a9\") "
Oct 30 22:22:48.159355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.159309    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-openshift-ca" pod "2fc66158-5167-46a8-8191-a0f3ea99d8a9" (UID: "2fc66158-5167-46a8-8191-a0f3ea99d8a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:48.159541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.159514    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:48.159541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.159542    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:48.161646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.161615    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-xks-ca" pod "2fc66158-5167-46a8-8191-a0f3ea99d8a9" (UID: "2fc66158-5167-46a8-8191-a0f3ea99d8a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:48.161767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.161683    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91279279-1fb9-49d7-bc82-b1519770631b-kube-api-access-hc9vl" pod "91279279-1fb9-49d7-bc82-b1519770631b" (UID: "91279279-1fb9-49d7-bc82-b1519770631b"). InnerVolumeSpecName "kube-api-access-hc9vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:48.162551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.162519    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-xks-ca" pod "91279279-1fb9-49d7-bc82-b1519770631b" (UID: "91279279-1fb9-49d7-bc82-b1519770631b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:48.163407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.163382    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc66158-5167-46a8-8191-a0f3ea99d8a9-kube-api-access-mjs8r" pod "2fc66158-5167-46a8-8191-a0f3ea99d8a9" (UID: "2fc66158-5167-46a8-8191-a0f3ea99d8a9"). InnerVolumeSpecName "kube-api-access-mjs8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:48.260942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.260898    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91279279-1fb9-49d7-bc82-b1519770631b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:48.260942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.260930    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc66158-5167-46a8-8191-a0f3ea99d8a9-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:48.260942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.260940    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjs8r\" (UniqueName: \"kubernetes.io/projected/2fc66158-5167-46a8-8191-a0f3ea99d8a9-kube-api-access-mjs8r\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:48.260942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.260951    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc9vl\" (UniqueName: \"kubernetes.io/projected/91279279-1fb9-49d7-bc82-b1519770631b-kube-api-access-hc9vl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:48.403349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.403303    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:48.405034 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.405008    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:48.407395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.407374    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:22:48.696584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.696478    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f" event={"ID":"91279279-1fb9-49d7-bc82-b1519770631b","Type":"ContainerDied","Data":"41ee9ac5d7a4eebcc0e75b73cdd1f434d498714fabd8efc7ce46c96f2224f409"}
Oct 30 22:22:48.696584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.696542    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"
Oct 30 22:22:48.696584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.696561    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41ee9ac5d7a4eebcc0e75b73cdd1f434d498714fabd8efc7ce46c96f2224f409"
Oct 30 22:22:48.698325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.698305    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"
Oct 30 22:22:48.698456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.698348    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm" event={"ID":"2fc66158-5167-46a8-8191-a0f3ea99d8a9","Type":"ContainerDied","Data":"da5162ae6d3b6b3bd508de5c66d0c9ce2f2486a0f4b97e8eb1f2a5ae0af6aada"}
Oct 30 22:22:48.698456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:48.698400    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da5162ae6d3b6b3bd508de5c66d0c9ce2f2486a0f4b97e8eb1f2a5ae0af6aada"
Oct 30 22:22:50.668519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.668480    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"]
Oct 30 22:22:50.669182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669165    4581 cpu_manager.go:414] "containerMap: removing container" podUID="91279279-1fb9-49d7-bc82-b1519770631b" containerName="revoke-keys"
Oct 30 22:22:50.669182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669182    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91279279-1fb9-49d7-bc82-b1519770631b" containerName="revoke-keys"
Oct 30 22:22:50.669383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669196    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2fc66158-5167-46a8-8191-a0f3ea99d8a9" containerName="revoke-keys"
Oct 30 22:22:50.669383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669201    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc66158-5167-46a8-8191-a0f3ea99d8a9" containerName="revoke-keys"
Oct 30 22:22:50.669383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669239    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3" containerName="revoke-keys"
Oct 30 22:22:50.669383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669244    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3" containerName="revoke-keys"
Oct 30 22:22:50.669383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669350    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="91279279-1fb9-49d7-bc82-b1519770631b" containerName="revoke-keys"
Oct 30 22:22:50.669383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669360    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc66158-5167-46a8-8191-a0f3ea99d8a9" containerName="revoke-keys"
Oct 30 22:22:50.669383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.669370    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3" containerName="revoke-keys"
Oct 30 22:22:50.673517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.673493    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.696841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.696188    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:50.696841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.696238    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:50.696841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.696253    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:22:50.696841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.696289    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:50.706056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.706022    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"]
Oct 30 22:22:50.710795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.710737    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.710962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.710932    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.711063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.710990    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svz6\" (UniqueName: \"kubernetes.io/projected/417c9962-267d-42b2-a881-2806ae57b9cc-kube-api-access-4svz6\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.812739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.812705    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svz6\" (UniqueName: \"kubernetes.io/projected/417c9962-267d-42b2-a881-2806ae57b9cc-kube-api-access-4svz6\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.812909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.812894    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.813049 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.813034    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.813618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.813591    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.815601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.815579    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.830607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.830580    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svz6\" (UniqueName: \"kubernetes.io/projected/417c9962-267d-42b2-a881-2806ae57b9cc-kube-api-access-4svz6\") pod \"maas-api-revoke-sub-c720e262-kh4wl\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:50.992774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:50.992692    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:51.313944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:51.313912    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"]
Oct 30 22:22:51.314951 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:51.314922    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417c9962_267d_42b2_a881_2806ae57b9cc.slice/crio-387cfd89b874473ba3424ff88dc6e61d9caadf7d06599af20c0122496eb1e2bc WatchSource:0}: Error finding container 387cfd89b874473ba3424ff88dc6e61d9caadf7d06599af20c0122496eb1e2bc: Status 404 returned error can't find the container with id 387cfd89b874473ba3424ff88dc6e61d9caadf7d06599af20c0122496eb1e2bc
Oct 30 22:22:51.747668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:51.747629    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl" event={"ID":"417c9962-267d-42b2-a881-2806ae57b9cc","Type":"ContainerStarted","Data":"726d6521fade1f657d1941f148aa605401e43c619bab71459740a11d518c1896"}
Oct 30 22:22:51.748134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:51.747684    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl" event={"ID":"417c9962-267d-42b2-a881-2806ae57b9cc","Type":"ContainerStarted","Data":"387cfd89b874473ba3424ff88dc6e61d9caadf7d06599af20c0122496eb1e2bc"}
Oct 30 22:22:51.787870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:51.787800    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl" podStartSLOduration=1.787782084 podStartE2EDuration="1.787782084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:51.783607114 +0000 UTC m=+955.822546001" watchObservedRunningTime="2026-10-30 22:22:51.787782084 +0000 UTC m=+955.826720972"
Oct 30 22:22:52.762162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:52.762126    4581 generic.go:356] "Generic (PLEG): container finished" podID="417c9962-267d-42b2-a881-2806ae57b9cc" containerID="726d6521fade1f657d1941f148aa605401e43c619bab71459740a11d518c1896" exitCode=0
Oct 30 22:22:52.762601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:52.762230    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl" event={"ID":"417c9962-267d-42b2-a881-2806ae57b9cc","Type":"ContainerDied","Data":"726d6521fade1f657d1941f148aa605401e43c619bab71459740a11d518c1896"}
Oct 30 22:22:53.170862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.170820    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"]
Oct 30 22:22:53.176879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.176841    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.192558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.192526    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"]
Oct 30 22:22:53.250179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.250141    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5756x\" (UniqueName: \"kubernetes.io/projected/71dc3843-fab9-4abb-b27f-b3d96a72f08e-kube-api-access-5756x\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.250448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.250419    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.250557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.250540    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.352590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.352552    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5756x\" (UniqueName: \"kubernetes.io/projected/71dc3843-fab9-4abb-b27f-b3d96a72f08e-kube-api-access-5756x\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.352838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.352770    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.352900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.352848    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.353619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.353590    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.356187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.356159    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.372359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.372327    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5756x\" (UniqueName: \"kubernetes.io/projected/71dc3843-fab9-4abb-b27f-b3d96a72f08e-kube-api-access-5756x\") pod \"maas-api-revoke-sub-d8b80472-479lq\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.400280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.400244    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"]
Oct 30 22:22:53.405290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.405269    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.424304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.424207    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"]
Oct 30 22:22:53.454157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.454120    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.454375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.454351    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b889\" (UniqueName: \"kubernetes.io/projected/e0ea692e-aea3-499e-b747-f72d9d901aaf-kube-api-access-9b889\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.454502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.454484    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.503059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.503025    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:53.555790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.555753    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.555981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.555911    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b889\" (UniqueName: \"kubernetes.io/projected/e0ea692e-aea3-499e-b747-f72d9d901aaf-kube-api-access-9b889\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.556049 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.556038    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.557031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.557005    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.559677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.559628    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.582429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.578439    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b889\" (UniqueName: \"kubernetes.io/projected/e0ea692e-aea3-499e-b747-f72d9d901aaf-kube-api-access-9b889\") pod \"maas-api-revoke-sub-598f4cc7-r2k7n\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.723720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.723636    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:53.834245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:53.833876    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"]
Oct 30 22:22:54.062947 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:54.062904    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ea692e_aea3_499e_b747_f72d9d901aaf.slice/crio-1a456d90ad5ab6b6686c92b979f1e1d7c413ccb3e74d3881afddfd5ee4b07f86 WatchSource:0}: Error finding container 1a456d90ad5ab6b6686c92b979f1e1d7c413ccb3e74d3881afddfd5ee4b07f86: Status 404 returned error can't find the container with id 1a456d90ad5ab6b6686c92b979f1e1d7c413ccb3e74d3881afddfd5ee4b07f86
Oct 30 22:22:54.066355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.066333    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:54.067255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.067187    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"]
Oct 30 22:22:54.169169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.169143    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-openshift-ca\") pod \"417c9962-267d-42b2-a881-2806ae57b9cc\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") "
Oct 30 22:22:54.169387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.169367    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-xks-ca\") pod \"417c9962-267d-42b2-a881-2806ae57b9cc\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") "
Oct 30 22:22:54.169534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.169520    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417c9962-267d-42b2-a881-2806ae57b9cc-kube-api-access-4svz6\" (UniqueName: \"kubernetes.io/projected/417c9962-267d-42b2-a881-2806ae57b9cc-kube-api-access-4svz6\") pod \"417c9962-267d-42b2-a881-2806ae57b9cc\" (UID: \"417c9962-267d-42b2-a881-2806ae57b9cc\") "
Oct 30 22:22:54.170899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.170704    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-openshift-ca" pod "417c9962-267d-42b2-a881-2806ae57b9cc" (UID: "417c9962-267d-42b2-a881-2806ae57b9cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:54.276783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.275896    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:54.352025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.351906    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417c9962-267d-42b2-a881-2806ae57b9cc-kube-api-access-4svz6" pod "417c9962-267d-42b2-a881-2806ae57b9cc" (UID: "417c9962-267d-42b2-a881-2806ae57b9cc"). InnerVolumeSpecName "kube-api-access-4svz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:54.352421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.352383    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-xks-ca" pod "417c9962-267d-42b2-a881-2806ae57b9cc" (UID: "417c9962-267d-42b2-a881-2806ae57b9cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:54.378277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.378239    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417c9962-267d-42b2-a881-2806ae57b9cc-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:54.378277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.378281    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4svz6\" (UniqueName: \"kubernetes.io/projected/417c9962-267d-42b2-a881-2806ae57b9cc-kube-api-access-4svz6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:54.788858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.788821    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n" event={"ID":"e0ea692e-aea3-499e-b747-f72d9d901aaf","Type":"ContainerStarted","Data":"5d290744323411cb679e09ff532dc05ccdec6b9e3120cf4dd0c95576b2c329ee"}
Oct 30 22:22:54.789119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.788899    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n" event={"ID":"e0ea692e-aea3-499e-b747-f72d9d901aaf","Type":"ContainerStarted","Data":"1a456d90ad5ab6b6686c92b979f1e1d7c413ccb3e74d3881afddfd5ee4b07f86"}
Oct 30 22:22:54.791472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.791429    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl" event={"ID":"417c9962-267d-42b2-a881-2806ae57b9cc","Type":"ContainerDied","Data":"387cfd89b874473ba3424ff88dc6e61d9caadf7d06599af20c0122496eb1e2bc"}
Oct 30 22:22:54.791619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.791492    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="387cfd89b874473ba3424ff88dc6e61d9caadf7d06599af20c0122496eb1e2bc"
Oct 30 22:22:54.791619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.791439    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"
Oct 30 22:22:54.793167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.793142    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq" event={"ID":"71dc3843-fab9-4abb-b27f-b3d96a72f08e","Type":"ContainerStarted","Data":"676bfa4f452877700281e94de7113c088e0e14655e2aec5bd06cff4cc12396ce"}
Oct 30 22:22:54.793335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.793187    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq" event={"ID":"71dc3843-fab9-4abb-b27f-b3d96a72f08e","Type":"ContainerStarted","Data":"23d714b89d5f2e37db2039233d932604645eb6fb381e377e7088d417958bc06e"}
Oct 30 22:22:54.832457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.832404    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n" podStartSLOduration=1.832388941 podStartE2EDuration="1.832388941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:54.824642836 +0000 UTC m=+958.863581721" watchObservedRunningTime="2026-10-30 22:22:54.832388941 +0000 UTC m=+958.871327827"
Oct 30 22:22:54.880536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:54.880465    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq" podStartSLOduration=1.88044425 podStartE2EDuration="1.88044425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:54.877074362 +0000 UTC m=+958.916013249" watchObservedRunningTime="2026-10-30 22:22:54.88044425 +0000 UTC m=+958.919383137"
Oct 30 22:22:55.804666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:55.804625    4581 generic.go:356] "Generic (PLEG): container finished" podID="e0ea692e-aea3-499e-b747-f72d9d901aaf" containerID="5d290744323411cb679e09ff532dc05ccdec6b9e3120cf4dd0c95576b2c329ee" exitCode=0
Oct 30 22:22:55.804883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:55.804701    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n" event={"ID":"e0ea692e-aea3-499e-b747-f72d9d901aaf","Type":"ContainerDied","Data":"5d290744323411cb679e09ff532dc05ccdec6b9e3120cf4dd0c95576b2c329ee"}
Oct 30 22:22:55.806640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:55.806616    4581 generic.go:356] "Generic (PLEG): container finished" podID="71dc3843-fab9-4abb-b27f-b3d96a72f08e" containerID="676bfa4f452877700281e94de7113c088e0e14655e2aec5bd06cff4cc12396ce" exitCode=0
Oct 30 22:22:55.806773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:55.806695    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq" event={"ID":"71dc3843-fab9-4abb-b27f-b3d96a72f08e","Type":"ContainerDied","Data":"676bfa4f452877700281e94de7113c088e0e14655e2aec5bd06cff4cc12396ce"}
Oct 30 22:22:56.535558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.535523    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"]
Oct 30 22:22:56.536485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.536461    4581 cpu_manager.go:414] "containerMap: removing container" podUID="417c9962-267d-42b2-a881-2806ae57b9cc" containerName="revoke-keys"
Oct 30 22:22:56.536485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.536484    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417c9962-267d-42b2-a881-2806ae57b9cc" containerName="revoke-keys"
Oct 30 22:22:56.536625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.536614    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="417c9962-267d-42b2-a881-2806ae57b9cc" containerName="revoke-keys"
Oct 30 22:22:56.540559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.540543    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.563089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.563055    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"]
Oct 30 22:22:56.616482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.616447    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.616701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.616638    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.616701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.616683    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhn7s\" (UniqueName: \"kubernetes.io/projected/7f455c4b-b4ea-4af4-8076-a900a6627523-kube-api-access-xhn7s\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.718393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.718354    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.718606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.718557    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.718606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.718601    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhn7s\" (UniqueName: \"kubernetes.io/projected/7f455c4b-b4ea-4af4-8076-a900a6627523-kube-api-access-xhn7s\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.719331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.719307    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.721046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.721022    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.737456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.737425    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhn7s\" (UniqueName: \"kubernetes.io/projected/7f455c4b-b4ea-4af4-8076-a900a6627523-kube-api-access-xhn7s\") pod \"maas-api-revoke-sub-1ae9919f-n9hv8\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:56.858015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:56.857265    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:22:57.050613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.050200    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:57.062422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.062394    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:57.133266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.130310    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ea692e-aea3-499e-b747-f72d9d901aaf-kube-api-access-9b889\" (UniqueName: \"kubernetes.io/projected/e0ea692e-aea3-499e-b747-f72d9d901aaf-kube-api-access-9b889\") pod \"e0ea692e-aea3-499e-b747-f72d9d901aaf\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") "
Oct 30 22:22:57.133266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.130432    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-openshift-ca\") pod \"e0ea692e-aea3-499e-b747-f72d9d901aaf\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") "
Oct 30 22:22:57.133266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.130483    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-openshift-ca\") pod \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") "
Oct 30 22:22:57.133266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.130583    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-xks-ca\") pod \"e0ea692e-aea3-499e-b747-f72d9d901aaf\" (UID: \"e0ea692e-aea3-499e-b747-f72d9d901aaf\") "
Oct 30 22:22:57.133266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.130618    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71dc3843-fab9-4abb-b27f-b3d96a72f08e-kube-api-access-5756x\" (UniqueName: \"kubernetes.io/projected/71dc3843-fab9-4abb-b27f-b3d96a72f08e-kube-api-access-5756x\") pod \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") "
Oct 30 22:22:57.133266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.130670    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-xks-ca\") pod \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\" (UID: \"71dc3843-fab9-4abb-b27f-b3d96a72f08e\") "
Oct 30 22:22:57.133266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.132576    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-openshift-ca" pod "71dc3843-fab9-4abb-b27f-b3d96a72f08e" (UID: "71dc3843-fab9-4abb-b27f-b3d96a72f08e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:57.133266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.132653    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-openshift-ca" pod "e0ea692e-aea3-499e-b747-f72d9d901aaf" (UID: "e0ea692e-aea3-499e-b747-f72d9d901aaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:57.137783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.136882    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-xks-ca" pod "e0ea692e-aea3-499e-b747-f72d9d901aaf" (UID: "e0ea692e-aea3-499e-b747-f72d9d901aaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:57.138167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.138142    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dc3843-fab9-4abb-b27f-b3d96a72f08e-kube-api-access-5756x" pod "71dc3843-fab9-4abb-b27f-b3d96a72f08e" (UID: "71dc3843-fab9-4abb-b27f-b3d96a72f08e"). InnerVolumeSpecName "kube-api-access-5756x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:57.138437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.138402    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ea692e-aea3-499e-b747-f72d9d901aaf-kube-api-access-9b889" pod "e0ea692e-aea3-499e-b747-f72d9d901aaf" (UID: "e0ea692e-aea3-499e-b747-f72d9d901aaf"). InnerVolumeSpecName "kube-api-access-9b889". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:57.143063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.143034    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-xks-ca" pod "71dc3843-fab9-4abb-b27f-b3d96a72f08e" (UID: "71dc3843-fab9-4abb-b27f-b3d96a72f08e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:57.215775 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:57.215743    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f455c4b_b4ea_4af4_8076_a900a6627523.slice/crio-f122bb39450f4471289e142674587c5ea94440eb2d881c0bb1ee0aa639fdda46 WatchSource:0}: Error finding container f122bb39450f4471289e142674587c5ea94440eb2d881c0bb1ee0aa639fdda46: Status 404 returned error can't find the container with id f122bb39450f4471289e142674587c5ea94440eb2d881c0bb1ee0aa639fdda46
Oct 30 22:22:57.216074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.216053    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"]
Oct 30 22:22:57.232730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.232703    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:57.232860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.232733    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5756x\" (UniqueName: \"kubernetes.io/projected/71dc3843-fab9-4abb-b27f-b3d96a72f08e-kube-api-access-5756x\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:57.232860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.232748    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:57.232860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.232764    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b889\" (UniqueName: \"kubernetes.io/projected/e0ea692e-aea3-499e-b747-f72d9d901aaf-kube-api-access-9b889\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:57.232860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.232776    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea692e-aea3-499e-b747-f72d9d901aaf-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:57.232860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.232788    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc3843-fab9-4abb-b27f-b3d96a72f08e-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:22:57.511650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.511608    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"]
Oct 30 22:22:57.512771 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.512747    4581 cpu_manager.go:414] "containerMap: removing container" podUID="71dc3843-fab9-4abb-b27f-b3d96a72f08e" containerName="revoke-keys"
Oct 30 22:22:57.512830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.512776    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71dc3843-fab9-4abb-b27f-b3d96a72f08e" containerName="revoke-keys"
Oct 30 22:22:57.512872 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.512838    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e0ea692e-aea3-499e-b747-f72d9d901aaf" containerName="revoke-keys"
Oct 30 22:22:57.512872 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.512847    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ea692e-aea3-499e-b747-f72d9d901aaf" containerName="revoke-keys"
Oct 30 22:22:57.513028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.513013    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ea692e-aea3-499e-b747-f72d9d901aaf" containerName="revoke-keys"
Oct 30 22:22:57.513069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.513041    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="71dc3843-fab9-4abb-b27f-b3d96a72f08e" containerName="revoke-keys"
Oct 30 22:22:57.519304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.519273    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.538844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.538657    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"]
Oct 30 22:22:57.641135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.641098    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.641135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.641143    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.641374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.641329    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rcp\" (UniqueName: \"kubernetes.io/projected/973ff3e1-f903-4c36-ab13-4d5a2057019d-kube-api-access-85rcp\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.742534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.742496    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.742696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.742550    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.742696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.742652    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rcp\" (UniqueName: \"kubernetes.io/projected/973ff3e1-f903-4c36-ab13-4d5a2057019d-kube-api-access-85rcp\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.743554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.743528    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.745282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.745252    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.761495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.761463    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rcp\" (UniqueName: \"kubernetes.io/projected/973ff3e1-f903-4c36-ab13-4d5a2057019d-kube-api-access-85rcp\") pod \"maas-api-revoke-sub-f86019a1-klp2b\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.826314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.826273    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8" event={"ID":"7f455c4b-b4ea-4af4-8076-a900a6627523","Type":"ContainerStarted","Data":"be63fc1e66c418f0a51a4b01fe546158357f311c983ce10e90663f827ab779ce"}
Oct 30 22:22:57.826519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.826334    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8" event={"ID":"7f455c4b-b4ea-4af4-8076-a900a6627523","Type":"ContainerStarted","Data":"f122bb39450f4471289e142674587c5ea94440eb2d881c0bb1ee0aa639fdda46"}
Oct 30 22:22:57.827835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.827812    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"
Oct 30 22:22:57.827986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.827861    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq" event={"ID":"71dc3843-fab9-4abb-b27f-b3d96a72f08e","Type":"ContainerDied","Data":"23d714b89d5f2e37db2039233d932604645eb6fb381e377e7088d417958bc06e"}
Oct 30 22:22:57.827986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.827909    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23d714b89d5f2e37db2039233d932604645eb6fb381e377e7088d417958bc06e"
Oct 30 22:22:57.829734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.829712    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"
Oct 30 22:22:57.829835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.829793    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n" event={"ID":"e0ea692e-aea3-499e-b747-f72d9d901aaf","Type":"ContainerDied","Data":"1a456d90ad5ab6b6686c92b979f1e1d7c413ccb3e74d3881afddfd5ee4b07f86"}
Oct 30 22:22:57.829902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.829834    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a456d90ad5ab6b6686c92b979f1e1d7c413ccb3e74d3881afddfd5ee4b07f86"
Oct 30 22:22:57.844456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.844435    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:22:57.866564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:57.866505    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8" podStartSLOduration=1.8664865179999999 podStartE2EDuration="1.866486518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:57.859931486 +0000 UTC m=+961.898870373" watchObservedRunningTime="2026-10-30 22:22:57.866486518 +0000 UTC m=+961.905425405"
Oct 30 22:22:58.196709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:58.190975    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"]
Oct 30 22:22:58.196709 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:22:58.196368    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973ff3e1_f903_4c36_ab13_4d5a2057019d.slice/crio-b66240f433d385ccfa99bdc4dbe9361bbe98ebcd6768468ef45f2174d884a3ba WatchSource:0}: Error finding container b66240f433d385ccfa99bdc4dbe9361bbe98ebcd6768468ef45f2174d884a3ba: Status 404 returned error can't find the container with id b66240f433d385ccfa99bdc4dbe9361bbe98ebcd6768468ef45f2174d884a3ba
Oct 30 22:22:58.895237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:58.889656    4581 generic.go:356] "Generic (PLEG): container finished" podID="7f455c4b-b4ea-4af4-8076-a900a6627523" containerID="be63fc1e66c418f0a51a4b01fe546158357f311c983ce10e90663f827ab779ce" exitCode=0
Oct 30 22:22:58.895237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:58.889791    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8" event={"ID":"7f455c4b-b4ea-4af4-8076-a900a6627523","Type":"ContainerDied","Data":"be63fc1e66c418f0a51a4b01fe546158357f311c983ce10e90663f827ab779ce"}
Oct 30 22:22:58.916313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:58.915257    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b" event={"ID":"973ff3e1-f903-4c36-ab13-4d5a2057019d","Type":"ContainerStarted","Data":"60b8bc7f1fba14003e462f0eb5fba160ec1040da68d24439ae2b9fd843d3b058"}
Oct 30 22:22:58.916313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:58.915329    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b" event={"ID":"973ff3e1-f903-4c36-ab13-4d5a2057019d","Type":"ContainerStarted","Data":"b66240f433d385ccfa99bdc4dbe9361bbe98ebcd6768468ef45f2174d884a3ba"}
Oct 30 22:22:59.840312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:59.703188    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b" podStartSLOduration=2.703166027 podStartE2EDuration="2.703166027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:59.019855493 +0000 UTC m=+963.058794379" watchObservedRunningTime="2026-10-30 22:22:59.703166027 +0000 UTC m=+963.742104914"
Oct 30 22:22:59.999250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:59.997875    4581 generic.go:356] "Generic (PLEG): container finished" podID="973ff3e1-f903-4c36-ab13-4d5a2057019d" containerID="60b8bc7f1fba14003e462f0eb5fba160ec1040da68d24439ae2b9fd843d3b058" exitCode=0
Oct 30 22:22:59.999250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:22:59.998923    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b" event={"ID":"973ff3e1-f903-4c36-ab13-4d5a2057019d","Type":"ContainerDied","Data":"60b8bc7f1fba14003e462f0eb5fba160ec1040da68d24439ae2b9fd843d3b058"}
Oct 30 22:23:00.263563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.256949    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:23:00.411321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.411258    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.444260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.422532    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.465295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.464439    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.637385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.637331    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.725245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.724355    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:00.789378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.789324    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.817247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.815336    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:00.860727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:00.860622    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.163873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.163824    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.207737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.188991    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.275244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.274359    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.275244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.275235    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:23:01.437481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.437370    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:23:01.479513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.448820    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:23:01.505384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.500689    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:23:01.505384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.500763    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:23:01.576289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.560713    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:23:01.729313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.578349    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:23:01.729313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.613476    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:23:01.729313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.656377    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.729313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.657166    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:23:01.758320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.739537    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.758320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.740633    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:23:01.819288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.803583    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.819288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.804731    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:23:01.833139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.833096    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:01.871444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.870608    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:23:01.871444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.834805    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded"
Oct 30 22:23:01.918464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.918428    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:01.918733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:01.918705    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.014306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.012826    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.196290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.195779    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.238440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.214131    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.386298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.384455    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.396255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.387655    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.396807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.396766    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.430784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.430744    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.431032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.431000    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.447248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.447107    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.449435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.449403    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.508243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.499993    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:23:02.508243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.500055    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:23:02.589243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.588724    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.790247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.767848    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.792396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.792351    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.825590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.818356    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.841278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.837418    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.865404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.865350    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.972247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.971588    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.972247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:02.971652    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.004309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.000774    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:23:03.030427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.009028    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.030838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.030796    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.201425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.201373    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.352535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.224580    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.367981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.254281    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.399431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.305027    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:23:03.399431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.333357    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.399431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.381244    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.424695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.424332    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.447779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.447735    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.449643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.449608    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f455c4b-b4ea-4af4-8076-a900a6627523-kube-api-access-xhn7s\" (UniqueName: \"kubernetes.io/projected/7f455c4b-b4ea-4af4-8076-a900a6627523-kube-api-access-xhn7s\") pod \"7f455c4b-b4ea-4af4-8076-a900a6627523\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") "
Oct 30 22:23:03.449922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.449901    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-xks-ca\") pod \"7f455c4b-b4ea-4af4-8076-a900a6627523\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") "
Oct 30 22:23:03.469454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.469415    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-openshift-ca\") pod \"7f455c4b-b4ea-4af4-8076-a900a6627523\" (UID: \"7f455c4b-b4ea-4af4-8076-a900a6627523\") "
Oct 30 22:23:03.471059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.471026    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.602816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.550870    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.602816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.550924    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.603158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.603112    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.767304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.708328    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.767304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.708402    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.767304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.730150    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f455c4b-b4ea-4af4-8076-a900a6627523-kube-api-access-xhn7s" pod "7f455c4b-b4ea-4af4-8076-a900a6627523" (UID: "7f455c4b-b4ea-4af4-8076-a900a6627523"). InnerVolumeSpecName "kube-api-access-xhn7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:03.769591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.769553    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-openshift-ca" pod "7f455c4b-b4ea-4af4-8076-a900a6627523" (UID: "7f455c4b-b4ea-4af4-8076-a900a6627523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:03.849552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.849405    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.880127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.870673    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.880127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.872157    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.880127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.872495    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.916252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.887965    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhn7s\" (UniqueName: \"kubernetes.io/projected/7f455c4b-b4ea-4af4-8076-a900a6627523-kube-api-access-xhn7s\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:03.930332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.930291    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:03.942431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:03.888616    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-xks-ca" pod "7f455c4b-b4ea-4af4-8076-a900a6627523" (UID: "7f455c4b-b4ea-4af4-8076-a900a6627523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:04.032637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.032592    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8" event={"ID":"7f455c4b-b4ea-4af4-8076-a900a6627523","Type":"ContainerDied","Data":"f122bb39450f4471289e142674587c5ea94440eb2d881c0bb1ee0aa639fdda46"}
Oct 30 22:23:04.091421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.091384    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f122bb39450f4471289e142674587c5ea94440eb2d881c0bb1ee0aa639fdda46"
Oct 30 22:23:04.121251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.087640    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"
Oct 30 22:23:04.364978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.364933    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.410364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.378622    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.434253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.378883    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.452259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.420836    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" podUID="b7e5b659-2162-4552-919d-40c9866fd577" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.33:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.476374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.453146    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.578255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.576793    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.578255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.577710    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.622720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.622614    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.771520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.771454    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": context deadline exceeded"
Oct 30 22:23:04.894306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.885899    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" podUID="b7e5b659-2162-4552-919d-40c9866fd577" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.33:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.934418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:04.934368    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.204306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.187405    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.204306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.187761    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.204306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.187977    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.394492 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:23:05.394458    4581 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.251s"
Oct 30 22:23:05.453431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.453394    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.480511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.465499    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.499246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.484032    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.850391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.834144    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.865247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.861457    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.877324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.876816    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f455c4b-b4ea-4af4-8076-a900a6627523-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:05.877324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.877029    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.941248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.939424    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.013314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.012629    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp" podUID="90d4ed54-8bcf-442b-82c4-cbae47556237" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.043244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:05.948391    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.195250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.192360    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.202446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.202391    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:06.369249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.202945    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.485344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.482766    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.570245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.500927    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.570245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.502244    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.768932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.768840    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:23:06.837276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.835673    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:06.877413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.869873    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:23:06.878393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.878346    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:23:06.938620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.938580    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-openshift-ca\") pod \"973ff3e1-f903-4c36-ab13-4d5a2057019d\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") "
Oct 30 22:23:06.939010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.938983    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973ff3e1-f903-4c36-ab13-4d5a2057019d-kube-api-access-85rcp\" (UniqueName: \"kubernetes.io/projected/973ff3e1-f903-4c36-ab13-4d5a2057019d-kube-api-access-85rcp\") pod \"973ff3e1-f903-4c36-ab13-4d5a2057019d\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") "
Oct 30 22:23:07.217242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.205382    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-xks-ca\") pod \"973ff3e1-f903-4c36-ab13-4d5a2057019d\" (UID: \"973ff3e1-f903-4c36-ab13-4d5a2057019d\") "
Oct 30 22:23:07.217242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.939692    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-openshift-ca" pod "973ff3e1-f903-4c36-ab13-4d5a2057019d" (UID: "973ff3e1-f903-4c36-ab13-4d5a2057019d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:07.217242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.945356    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:07.217242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.206847    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.217242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:06.939846    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.217242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.038597    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.217242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.195308    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.276121 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:23:07.275399    4581 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f455c4b_b4ea_4af4_8076_a900a6627523.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:23:07.307598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.307556    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-xks-ca" pod "973ff3e1-f903-4c36-ab13-4d5a2057019d" (UID: "973ff3e1-f903-4c36-ab13-4d5a2057019d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:07.309000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.308971    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:07.309145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.309131    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973ff3e1-f903-4c36-ab13-4d5a2057019d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:07.342582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.342530    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.343321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.343288    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.381788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.381739    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.482548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.482457    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973ff3e1-f903-4c36-ab13-4d5a2057019d-kube-api-access-85rcp" pod "973ff3e1-f903-4c36-ab13-4d5a2057019d" (UID: "973ff3e1-f903-4c36-ab13-4d5a2057019d"). InnerVolumeSpecName "kube-api-access-85rcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:07.597846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.586589    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rcp\" (UniqueName: \"kubernetes.io/projected/973ff3e1-f903-4c36-ab13-4d5a2057019d-kube-api-access-85rcp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:07.749302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.746949    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.766280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.763188    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.766280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.763554    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.766280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.765074    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:23:07.800862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.800826    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:07.801156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.801127    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.848248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.842717    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.869777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.869733    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b" event={"ID":"973ff3e1-f903-4c36-ab13-4d5a2057019d","Type":"ContainerDied","Data":"b66240f433d385ccfa99bdc4dbe9361bbe98ebcd6768468ef45f2174d884a3ba"}
Oct 30 22:23:07.870062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.870040    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b66240f433d385ccfa99bdc4dbe9361bbe98ebcd6768468ef45f2174d884a3ba"
Oct 30 22:23:07.895245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.870409    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"
Oct 30 22:23:07.895605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.879318    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.895914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.879364    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:07.979305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.979268    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"]
Oct 30 22:23:07.997256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.980594    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7f455c4b-b4ea-4af4-8076-a900a6627523" containerName="revoke-keys"
Oct 30 22:23:07.997619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.997580    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f455c4b-b4ea-4af4-8076-a900a6627523" containerName="revoke-keys"
Oct 30 22:23:07.997837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.997821    4581 cpu_manager.go:414] "containerMap: removing container" podUID="973ff3e1-f903-4c36-ab13-4d5a2057019d" containerName="revoke-keys"
Oct 30 22:23:07.997937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.997924    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973ff3e1-f903-4c36-ab13-4d5a2057019d" containerName="revoke-keys"
Oct 30 22:23:07.998330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.998307    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="973ff3e1-f903-4c36-ab13-4d5a2057019d" containerName="revoke-keys"
Oct 30 22:23:07.998480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:07.998467    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f455c4b-b4ea-4af4-8076-a900a6627523" containerName="revoke-keys"
Oct 30 22:23:08.013809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.013730    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.038256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.038200    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:23:08.051085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.051037    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.051536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.051508    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.051746 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.051727    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjcbb\" (UniqueName: \"kubernetes.io/projected/9e3104be-1fbe-496f-b457-506d6b4f39fc-kube-api-access-rjcbb\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.052037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.038892    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:08.062814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.038939    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:08.064207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.039032    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:08.241432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.241376    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:23:08.249259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.242009    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:23:08.256255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.249868    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:23:08.342991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.323420    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.342991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.323533    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjcbb\" (UniqueName: \"kubernetes.io/projected/9e3104be-1fbe-496f-b457-506d6b4f39fc-kube-api-access-rjcbb\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.342991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.323711    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.350293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.344145    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.361289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.350765    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:08.476514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.461892    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:08.575485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.574753    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjcbb\" (UniqueName: \"kubernetes.io/projected/9e3104be-1fbe-496f-b457-506d6b4f39fc-kube-api-access-rjcbb\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.639336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.639304    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-119a68a7-qtrw6\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:08.691898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.691745    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"]
Oct 30 22:23:08.692117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.692101    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:23:08.692459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.692441    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:23:08.692732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.692517    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:23:08.694244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.693592    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:23:08.694244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.693662    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:23:08.711390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.711351    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"]
Oct 30 22:23:08.718013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.717980    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"]
Oct 30 22:23:08.718181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.718098    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.719256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.718968    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:23:08.720728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.720708    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:23:08.722500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.722161    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:23:08.722500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.722249    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:23:08.743338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.729379    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:23:08.754258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.752018    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:23:08.802239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.800412    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95bk2\" (UniqueName: \"kubernetes.io/projected/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-kube-api-access-95bk2\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.802239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.800567    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.802239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.801236    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.904750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.904660    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.904750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.904736    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95bk2\" (UniqueName: \"kubernetes.io/projected/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-kube-api-access-95bk2\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.904999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.904805    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.905503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.905481    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.907621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.907591    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.923415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.923383    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95bk2\" (UniqueName: \"kubernetes.io/projected/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-kube-api-access-95bk2\") pod \"maas-api-revoke-sub-48e3ec73-cqkpz\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:08.932658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:08.932627    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:09.090248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.086356    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:09.277122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.276003    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"]
Oct 30 22:23:09.444210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.444133    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"]
Oct 30 22:23:09.474546 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:23:09.474506    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1420a2_f6d7_466d_b91f_1a8b6171ebd1.slice/crio-df269ff995222d4b17e1fe9166daded2b0697618aecdcd5d406ec26e204a3f5e WatchSource:0}: Error finding container df269ff995222d4b17e1fe9166daded2b0697618aecdcd5d406ec26e204a3f5e: Status 404 returned error can't find the container with id df269ff995222d4b17e1fe9166daded2b0697618aecdcd5d406ec26e204a3f5e
Oct 30 22:23:09.912788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.912745    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz" event={"ID":"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1","Type":"ContainerStarted","Data":"454955ff9d6f95841260dcd7f4749dc64c07c34ffd9ecda2ee3c95dde34491eb"}
Oct 30 22:23:09.913291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.912846    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz" event={"ID":"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1","Type":"ContainerStarted","Data":"df269ff995222d4b17e1fe9166daded2b0697618aecdcd5d406ec26e204a3f5e"}
Oct 30 22:23:09.914634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.914608    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6" event={"ID":"9e3104be-1fbe-496f-b457-506d6b4f39fc","Type":"ContainerStarted","Data":"255277d4f74115ed93e6178bdcfedab10ddb6cecdf947941e3b6bf89aec5b0f5"}
Oct 30 22:23:09.914745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.914654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6" event={"ID":"9e3104be-1fbe-496f-b457-506d6b4f39fc","Type":"ContainerStarted","Data":"a470bf31d45783c5c8756f633e4e4386c8db7e11c8bcce344cb7e094a8ea9db3"}
Oct 30 22:23:09.956399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.956332    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz" podStartSLOduration=3.956314602 podStartE2EDuration="3.956314602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:09.953070221 +0000 UTC m=+973.992009106" watchObservedRunningTime="2026-10-30 22:23:09.956314602 +0000 UTC m=+973.995253488"
Oct 30 22:23:09.995139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:09.995065    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6" podStartSLOduration=4.995043183 podStartE2EDuration="4.995043183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:09.99149169 +0000 UTC m=+974.030430575" watchObservedRunningTime="2026-10-30 22:23:09.995043183 +0000 UTC m=+974.033982070"
Oct 30 22:23:10.820417 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:23:10.820327    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:23:10.925877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:10.925842    4581 generic.go:356] "Generic (PLEG): container finished" podID="9e3104be-1fbe-496f-b457-506d6b4f39fc" containerID="255277d4f74115ed93e6178bdcfedab10ddb6cecdf947941e3b6bf89aec5b0f5" exitCode=0
Oct 30 22:23:10.926326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:10.925934    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6" event={"ID":"9e3104be-1fbe-496f-b457-506d6b4f39fc","Type":"ContainerDied","Data":"255277d4f74115ed93e6178bdcfedab10ddb6cecdf947941e3b6bf89aec5b0f5"}
Oct 30 22:23:10.927825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:10.927800    4581 generic.go:356] "Generic (PLEG): container finished" podID="ef1420a2-f6d7-466d-b91f-1a8b6171ebd1" containerID="454955ff9d6f95841260dcd7f4749dc64c07c34ffd9ecda2ee3c95dde34491eb" exitCode=0
Oct 30 22:23:10.927925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:10.927880    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz" event={"ID":"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1","Type":"ContainerDied","Data":"454955ff9d6f95841260dcd7f4749dc64c07c34ffd9ecda2ee3c95dde34491eb"}
Oct 30 22:23:12.182904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.182873    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:12.186984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.186954    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:12.265102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.265008    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-openshift-ca\") pod \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") "
Oct 30 22:23:12.265433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.265407    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3104be-1fbe-496f-b457-506d6b4f39fc-kube-api-access-rjcbb\" (UniqueName: \"kubernetes.io/projected/9e3104be-1fbe-496f-b457-506d6b4f39fc-kube-api-access-rjcbb\") pod \"9e3104be-1fbe-496f-b457-506d6b4f39fc\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") "
Oct 30 22:23:12.265586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.265568    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-openshift-ca\") pod \"9e3104be-1fbe-496f-b457-506d6b4f39fc\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") "
Oct 30 22:23:12.265986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.265942    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-xks-ca\") pod \"9e3104be-1fbe-496f-b457-506d6b4f39fc\" (UID: \"9e3104be-1fbe-496f-b457-506d6b4f39fc\") "
Oct 30 22:23:12.266210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.266192    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-xks-ca\") pod \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") "
Oct 30 22:23:12.266376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.266360    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-kube-api-access-95bk2\" (UniqueName: \"kubernetes.io/projected/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-kube-api-access-95bk2\") pod \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\" (UID: \"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1\") "
Oct 30 22:23:12.268588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.268554    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-openshift-ca" pod "ef1420a2-f6d7-466d-b91f-1a8b6171ebd1" (UID: "ef1420a2-f6d7-466d-b91f-1a8b6171ebd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:12.271815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.270947    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-openshift-ca" pod "9e3104be-1fbe-496f-b457-506d6b4f39fc" (UID: "9e3104be-1fbe-496f-b457-506d6b4f39fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:12.275273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.273772    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-xks-ca" pod "ef1420a2-f6d7-466d-b91f-1a8b6171ebd1" (UID: "ef1420a2-f6d7-466d-b91f-1a8b6171ebd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:12.294452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.294409    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3104be-1fbe-496f-b457-506d6b4f39fc-kube-api-access-rjcbb" pod "9e3104be-1fbe-496f-b457-506d6b4f39fc" (UID: "9e3104be-1fbe-496f-b457-506d6b4f39fc"). InnerVolumeSpecName "kube-api-access-rjcbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:12.301987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.301924    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-kube-api-access-95bk2" pod "ef1420a2-f6d7-466d-b91f-1a8b6171ebd1" (UID: "ef1420a2-f6d7-466d-b91f-1a8b6171ebd1"). InnerVolumeSpecName "kube-api-access-95bk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:12.303493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.303463    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-xks-ca" pod "9e3104be-1fbe-496f-b457-506d6b4f39fc" (UID: "9e3104be-1fbe-496f-b457-506d6b4f39fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:12.369240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.369184    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:12.369240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.369230    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjcbb\" (UniqueName: \"kubernetes.io/projected/9e3104be-1fbe-496f-b457-506d6b4f39fc-kube-api-access-rjcbb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:12.369240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.369241    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:12.369240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.369251    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3104be-1fbe-496f-b457-506d6b4f39fc-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:12.369597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.369261    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:12.369597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.369271    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95bk2\" (UniqueName: \"kubernetes.io/projected/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1-kube-api-access-95bk2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:12.486571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.486532    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"]
Oct 30 22:23:12.487337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.487320    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ef1420a2-f6d7-466d-b91f-1a8b6171ebd1" containerName="revoke-keys"
Oct 30 22:23:12.487337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.487336    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1420a2-f6d7-466d-b91f-1a8b6171ebd1" containerName="revoke-keys"
Oct 30 22:23:12.487513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.487351    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9e3104be-1fbe-496f-b457-506d6b4f39fc" containerName="revoke-keys"
Oct 30 22:23:12.487513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.487356    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3104be-1fbe-496f-b457-506d6b4f39fc" containerName="revoke-keys"
Oct 30 22:23:12.487513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.487464    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1420a2-f6d7-466d-b91f-1a8b6171ebd1" containerName="revoke-keys"
Oct 30 22:23:12.487513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.487476    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3104be-1fbe-496f-b457-506d6b4f39fc" containerName="revoke-keys"
Oct 30 22:23:12.492431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.492407    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.511723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.511690    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"]
Oct 30 22:23:12.572122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.572021    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21816b3-807d-496b-b707-58dd440613de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.572392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.572367    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm2z5\" (UniqueName: \"kubernetes.io/projected/a21816b3-807d-496b-b707-58dd440613de-kube-api-access-rm2z5\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.572499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.572477    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21816b3-807d-496b-b707-58dd440613de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.673857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.673818    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21816b3-807d-496b-b707-58dd440613de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.674088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.674020    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm2z5\" (UniqueName: \"kubernetes.io/projected/a21816b3-807d-496b-b707-58dd440613de-kube-api-access-rm2z5\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.674088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.674066    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21816b3-807d-496b-b707-58dd440613de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.674807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.674758    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21816b3-807d-496b-b707-58dd440613de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.676905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.676876    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21816b3-807d-496b-b707-58dd440613de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.691930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.691897    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm2z5\" (UniqueName: \"kubernetes.io/projected/a21816b3-807d-496b-b707-58dd440613de-kube-api-access-rm2z5\") pod \"maas-api-revoke-sub-6c137beb-t49ck\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.812551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.812514    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:12.957630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.957597    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6" event={"ID":"9e3104be-1fbe-496f-b457-506d6b4f39fc","Type":"ContainerDied","Data":"a470bf31d45783c5c8756f633e4e4386c8db7e11c8bcce344cb7e094a8ea9db3"}
Oct 30 22:23:12.961780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.957656    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a470bf31d45783c5c8756f633e4e4386c8db7e11c8bcce344cb7e094a8ea9db3"
Oct 30 22:23:12.961780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.957659    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"
Oct 30 22:23:12.975131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.972098    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"
Oct 30 22:23:12.975131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.972201    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz" event={"ID":"ef1420a2-f6d7-466d-b91f-1a8b6171ebd1","Type":"ContainerDied","Data":"df269ff995222d4b17e1fe9166daded2b0697618aecdcd5d406ec26e204a3f5e"}
Oct 30 22:23:12.975131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:12.972312    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df269ff995222d4b17e1fe9166daded2b0697618aecdcd5d406ec26e204a3f5e"
Oct 30 22:23:13.122038 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:23:13.122005    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21816b3_807d_496b_b707_58dd440613de.slice/crio-24207fbf237e5bc59b33f2f678cabc8398e6e16c3e34ee546ec3106a94671ecf WatchSource:0}: Error finding container 24207fbf237e5bc59b33f2f678cabc8398e6e16c3e34ee546ec3106a94671ecf: Status 404 returned error can't find the container with id 24207fbf237e5bc59b33f2f678cabc8398e6e16c3e34ee546ec3106a94671ecf
Oct 30 22:23:13.122968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:13.122941    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"]
Oct 30 22:23:13.989327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:13.989288    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck" event={"ID":"a21816b3-807d-496b-b707-58dd440613de","Type":"ContainerStarted","Data":"c685ee6256b5aa6c554bee731c09fa8f9941f453e140badc1061af848f5542d3"}
Oct 30 22:23:13.989714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:13.989351    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck" event={"ID":"a21816b3-807d-496b-b707-58dd440613de","Type":"ContainerStarted","Data":"24207fbf237e5bc59b33f2f678cabc8398e6e16c3e34ee546ec3106a94671ecf"}
Oct 30 22:23:14.028263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.028190    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck" podStartSLOduration=2.028174334 podStartE2EDuration="2.028174334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:14.026106355 +0000 UTC m=+978.065045240" watchObservedRunningTime="2026-10-30 22:23:14.028174334 +0000 UTC m=+978.067113270"
Oct 30 22:23:14.075475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.075436    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"]
Oct 30 22:23:14.080433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.080411    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.105377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.105341    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"]
Oct 30 22:23:14.199147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.199105    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.199353 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.199337    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.199496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.199476    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797z5\" (UniqueName: \"kubernetes.io/projected/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-kube-api-access-797z5\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.301236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.301133    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.301406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.301293    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797z5\" (UniqueName: \"kubernetes.io/projected/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-kube-api-access-797z5\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.301406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.301341    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.302046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.302022    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.303908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.303882    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.318613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.318588    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-797z5\" (UniqueName: \"kubernetes.io/projected/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-kube-api-access-797z5\") pod \"maas-api-revoke-sub-f1749884-wh7dz\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.398916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.398882    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:14.713509 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:14.713471    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"]
Oct 30 22:23:14.713800 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:23:14.713777    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6840a624_d9f5_4dbe_87f7_37c5df9e5e35.slice/crio-671d55b34e77407a9df41d660f487194e109e7fb954bdfefecc3752bdc88e2ca WatchSource:0}: Error finding container 671d55b34e77407a9df41d660f487194e109e7fb954bdfefecc3752bdc88e2ca: Status 404 returned error can't find the container with id 671d55b34e77407a9df41d660f487194e109e7fb954bdfefecc3752bdc88e2ca
Oct 30 22:23:15.000828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:15.000783    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz" event={"ID":"6840a624-d9f5-4dbe-87f7-37c5df9e5e35","Type":"ContainerStarted","Data":"671d55b34e77407a9df41d660f487194e109e7fb954bdfefecc3752bdc88e2ca"}
Oct 30 22:23:15.002838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:15.002809    4581 generic.go:356] "Generic (PLEG): container finished" podID="a21816b3-807d-496b-b707-58dd440613de" containerID="c685ee6256b5aa6c554bee731c09fa8f9941f453e140badc1061af848f5542d3" exitCode=0
Oct 30 22:23:15.002974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:15.002898    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck" event={"ID":"a21816b3-807d-496b-b707-58dd440613de","Type":"ContainerDied","Data":"c685ee6256b5aa6c554bee731c09fa8f9941f453e140badc1061af848f5542d3"}
Oct 30 22:23:16.015573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.015520    4581 generic.go:356] "Generic (PLEG): container finished" podID="6840a624-d9f5-4dbe-87f7-37c5df9e5e35" containerID="248361ce54e3a089fc121b158586b4827f9ea0b786cb7cfe2b12573a6871a616" exitCode=0
Oct 30 22:23:16.015973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.015607    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz" event={"ID":"6840a624-d9f5-4dbe-87f7-37c5df9e5e35","Type":"ContainerDied","Data":"248361ce54e3a089fc121b158586b4827f9ea0b786cb7cfe2b12573a6871a616"}
Oct 30 22:23:16.240937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.240909    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:16.248069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.248032    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21816b3-807d-496b-b707-58dd440613de-kube-api-access-rm2z5\" (UniqueName: \"kubernetes.io/projected/a21816b3-807d-496b-b707-58dd440613de-kube-api-access-rm2z5\") pod \"a21816b3-807d-496b-b707-58dd440613de\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") "
Oct 30 22:23:16.248258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.248236    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21816b3-807d-496b-b707-58dd440613de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21816b3-807d-496b-b707-58dd440613de-maas-api-xks-ca\") pod \"a21816b3-807d-496b-b707-58dd440613de\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") "
Oct 30 22:23:16.248373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.248340    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21816b3-807d-496b-b707-58dd440613de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21816b3-807d-496b-b707-58dd440613de-maas-api-openshift-ca\") pod \"a21816b3-807d-496b-b707-58dd440613de\" (UID: \"a21816b3-807d-496b-b707-58dd440613de\") "
Oct 30 22:23:16.248738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.248702    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21816b3-807d-496b-b707-58dd440613de-maas-api-openshift-ca" pod "a21816b3-807d-496b-b707-58dd440613de" (UID: "a21816b3-807d-496b-b707-58dd440613de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:16.248944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.248926    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21816b3-807d-496b-b707-58dd440613de-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:16.250399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.250375    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21816b3-807d-496b-b707-58dd440613de-kube-api-access-rm2z5" pod "a21816b3-807d-496b-b707-58dd440613de" (UID: "a21816b3-807d-496b-b707-58dd440613de"). InnerVolumeSpecName "kube-api-access-rm2z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:16.250496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.250429    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21816b3-807d-496b-b707-58dd440613de-maas-api-xks-ca" pod "a21816b3-807d-496b-b707-58dd440613de" (UID: "a21816b3-807d-496b-b707-58dd440613de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:16.349950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.349913    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm2z5\" (UniqueName: \"kubernetes.io/projected/a21816b3-807d-496b-b707-58dd440613de-kube-api-access-rm2z5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:16.349950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:16.349946    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21816b3-807d-496b-b707-58dd440613de-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:17.026817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.026784    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"
Oct 30 22:23:17.027313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.026816    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck" event={"ID":"a21816b3-807d-496b-b707-58dd440613de","Type":"ContainerDied","Data":"24207fbf237e5bc59b33f2f678cabc8398e6e16c3e34ee546ec3106a94671ecf"}
Oct 30 22:23:17.027313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.026863    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24207fbf237e5bc59b33f2f678cabc8398e6e16c3e34ee546ec3106a94671ecf"
Oct 30 22:23:17.235733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.235707    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:17.269441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.269407    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-openshift-ca\") pod \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") "
Oct 30 22:23:17.269657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.269494    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-xks-ca\") pod \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") "
Oct 30 22:23:17.269774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.269757    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-kube-api-access-797z5\" (UniqueName: \"kubernetes.io/projected/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-kube-api-access-797z5\") pod \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\" (UID: \"6840a624-d9f5-4dbe-87f7-37c5df9e5e35\") "
Oct 30 22:23:17.269834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.269802    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-openshift-ca" pod "6840a624-d9f5-4dbe-87f7-37c5df9e5e35" (UID: "6840a624-d9f5-4dbe-87f7-37c5df9e5e35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:17.270511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.270486    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:17.272268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.272206    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-xks-ca" pod "6840a624-d9f5-4dbe-87f7-37c5df9e5e35" (UID: "6840a624-d9f5-4dbe-87f7-37c5df9e5e35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:17.273116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.273088    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-kube-api-access-797z5" pod "6840a624-d9f5-4dbe-87f7-37c5df9e5e35" (UID: "6840a624-d9f5-4dbe-87f7-37c5df9e5e35"). InnerVolumeSpecName "kube-api-access-797z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:17.371823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.371783    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:17.371823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.371813    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-797z5\" (UniqueName: \"kubernetes.io/projected/6840a624-d9f5-4dbe-87f7-37c5df9e5e35-kube-api-access-797z5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:17.977108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.977073    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"]
Oct 30 22:23:17.978106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.978088    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a21816b3-807d-496b-b707-58dd440613de" containerName="revoke-keys"
Oct 30 22:23:17.978162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.978109    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21816b3-807d-496b-b707-58dd440613de" containerName="revoke-keys"
Oct 30 22:23:17.978162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.978143    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6840a624-d9f5-4dbe-87f7-37c5df9e5e35" containerName="revoke-keys"
Oct 30 22:23:17.978162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.978151    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6840a624-d9f5-4dbe-87f7-37c5df9e5e35" containerName="revoke-keys"
Oct 30 22:23:17.978346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.978333    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21816b3-807d-496b-b707-58dd440613de" containerName="revoke-keys"
Oct 30 22:23:17.978388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.978353    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6840a624-d9f5-4dbe-87f7-37c5df9e5e35" containerName="revoke-keys"
Oct 30 22:23:17.983198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:17.983167    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.001765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.001734    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"]
Oct 30 22:23:18.044253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.042371    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"
Oct 30 22:23:18.044253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.042394    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz" event={"ID":"6840a624-d9f5-4dbe-87f7-37c5df9e5e35","Type":"ContainerDied","Data":"671d55b34e77407a9df41d660f487194e109e7fb954bdfefecc3752bdc88e2ca"}
Oct 30 22:23:18.044253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.042455    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="671d55b34e77407a9df41d660f487194e109e7fb954bdfefecc3752bdc88e2ca"
Oct 30 22:23:18.082017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.081978    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zz95\" (UniqueName: \"kubernetes.io/projected/a189dbde-a4e1-40f6-90a5-745153598f1d-kube-api-access-7zz95\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.082283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.082038    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.082283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.082259    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.184137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.184100    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zz95\" (UniqueName: \"kubernetes.io/projected/a189dbde-a4e1-40f6-90a5-745153598f1d-kube-api-access-7zz95\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.184337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.184166    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.184337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.184298    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.185364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.185329    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.187609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.187580    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.202849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.202799    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zz95\" (UniqueName: \"kubernetes.io/projected/a189dbde-a4e1-40f6-90a5-745153598f1d-kube-api-access-7zz95\") pod \"maas-api-revoke-sub-0268846d-c9spq\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.301765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.301682    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:18.623596 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:23:18.623564    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda189dbde_a4e1_40f6_90a5_745153598f1d.slice/crio-936796bdc68c6972195f8b0c3dba48e1510d325ef1c7bbdcc5043822b1cf520b WatchSource:0}: Error finding container 936796bdc68c6972195f8b0c3dba48e1510d325ef1c7bbdcc5043822b1cf520b: Status 404 returned error can't find the container with id 936796bdc68c6972195f8b0c3dba48e1510d325ef1c7bbdcc5043822b1cf520b
Oct 30 22:23:18.624353 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:18.624330    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"]
Oct 30 22:23:19.055900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.055861    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq" event={"ID":"a189dbde-a4e1-40f6-90a5-745153598f1d","Type":"ContainerStarted","Data":"936796bdc68c6972195f8b0c3dba48e1510d325ef1c7bbdcc5043822b1cf520b"}
Oct 30 22:23:19.271207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.271108    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"]
Oct 30 22:23:19.283236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.277051    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.302434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.302401    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ca259734-openshift-default-dockercfg-tm9rz\""
Oct 30 22:23:19.303643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.303614    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istiod-ca-cert\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.303844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.303821    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-envoy\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.304038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.304021    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-data\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.304256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.304236    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-certs\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.304496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.304476    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-socket\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.304613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.304596    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-credential-socket\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.304738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.304724    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-token\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.304991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.304972    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffxt\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-kube-api-access-rffxt\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.305152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.305137    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-podinfo\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.307919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.307895    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"]
Oct 30 22:23:19.425117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425073    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-token\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.425390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425252    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffxt\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-kube-api-access-rffxt\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.425390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425314    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-podinfo\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.425390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425366    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istiod-ca-cert\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.425556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425397    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-envoy\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.425556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425494    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-data\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.425652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425583    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-certs\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.425705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425681    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-socket\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.425753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.425712    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-credential-socket\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.426127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.426102    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-credential-socket\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.427579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.427552    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-certs\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.427782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.427764    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-data\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.438242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.428247    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istiod-ca-cert\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.438242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.428551    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-socket\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.472569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.461683    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-podinfo\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.482613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.482579    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-envoy\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.519240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.514967    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffxt\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-kube-api-access-rffxt\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.519940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.519914    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-token\") pod \"e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") " pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:19.616280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:19.616241    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:20.083637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.083005    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"]
Oct 30 22:23:20.090148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.086687    4581 generic.go:356] "Generic (PLEG): container finished" podID="a189dbde-a4e1-40f6-90a5-745153598f1d" containerID="ad4e53ca929d7c789edbba49c32612b2b39e12196c4f86a0de0e8049b9904171" exitCode=0
Oct 30 22:23:20.090148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.086777    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq" event={"ID":"a189dbde-a4e1-40f6-90a5-745153598f1d","Type":"ContainerDied","Data":"ad4e53ca929d7c789edbba49c32612b2b39e12196c4f86a0de0e8049b9904171"}
Oct 30 22:23:20.106981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.106339    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:20.106981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.106431    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:20.106981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.106473    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:20.137391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.130462    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"]
Oct 30 22:23:20.233569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.233479    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"]
Oct 30 22:23:20.233884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.233866    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.276245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.265885    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1ba9ae08-openshift-default-dockercfg-nnpzt\""
Oct 30 22:23:20.325244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.324730    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istiod-ca-cert\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.325244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.324829    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-certs\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.325244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.324892    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-token\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.331588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.327048    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-envoy\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.331588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.327090    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-credential-socket\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.331588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.327122    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-podinfo\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.331588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.327158    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-data\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.331588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.327270    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f45wt\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-kube-api-access-f45wt\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.331588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.327306    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-socket\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429378    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-credential-socket\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429442    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-podinfo\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429485    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-data\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429557    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f45wt\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-kube-api-access-f45wt\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429596    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-socket\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429745    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istiod-ca-cert\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429792    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-credential-socket\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429825    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-certs\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429885    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-token\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.429991    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-envoy\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.430047    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-socket\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.430305    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-certs\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.430879    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istiod-ca-cert\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.431080    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-data\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.433471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.432629    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-podinfo\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.450291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.447284    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-envoy\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.485354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.474936    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f45wt\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-kube-api-access-f45wt\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.485354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.481756    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-token\") pod \"e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") " pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:20.646242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:20.645641    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:21.027912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.027875    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"]
Oct 30 22:23:21.029734 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:23:21.028608    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09b6a1c_f6cb_45a6_a976_417dbbc90465.slice/crio-d2e693d0c1fb048cfb6d6192ee7cf6cf2e7724c8d501bdfeef9bdda16fa7fee0 WatchSource:0}: Error finding container d2e693d0c1fb048cfb6d6192ee7cf6cf2e7724c8d501bdfeef9bdda16fa7fee0: Status 404 returned error can't find the container with id d2e693d0c1fb048cfb6d6192ee7cf6cf2e7724c8d501bdfeef9bdda16fa7fee0
Oct 30 22:23:21.031650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.031608    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:21.031778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.031693    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:21.031778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.031736    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:21.110805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.110764    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" event={"ID":"8880fff5-a9b3-4522-a4cf-8bba135c5d99","Type":"ContainerStarted","Data":"6970c30ee638e8a4968447fab5ce43a2b48c01a4777a649005b3cb6e46b730eb"}
Oct 30 22:23:21.111161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.110834    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" event={"ID":"8880fff5-a9b3-4522-a4cf-8bba135c5d99","Type":"ContainerStarted","Data":"bcd80093dba468c4008362b8b035f774dde533dac3164df29485341d8c340758"}
Oct 30 22:23:21.112596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.112569    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" event={"ID":"e09b6a1c-f6cb-45a6-a976-417dbbc90465","Type":"ContainerStarted","Data":"d2e693d0c1fb048cfb6d6192ee7cf6cf2e7724c8d501bdfeef9bdda16fa7fee0"}
Oct 30 22:23:21.168167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.168091    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podStartSLOduration=2.168069679 podStartE2EDuration="2.168069679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:21.147836066 +0000 UTC m=+985.186774954" watchObservedRunningTime="2026-10-30 22:23:21.168069679 +0000 UTC m=+985.207008565"
Oct 30 22:23:21.362402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.362328    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:21.481723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.481677    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-xks-ca\") pod \"a189dbde-a4e1-40f6-90a5-745153598f1d\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") "
Oct 30 22:23:21.481903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.481776    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a189dbde-a4e1-40f6-90a5-745153598f1d-kube-api-access-7zz95\" (UniqueName: \"kubernetes.io/projected/a189dbde-a4e1-40f6-90a5-745153598f1d-kube-api-access-7zz95\") pod \"a189dbde-a4e1-40f6-90a5-745153598f1d\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") "
Oct 30 22:23:21.481903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.481858    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-openshift-ca\") pod \"a189dbde-a4e1-40f6-90a5-745153598f1d\" (UID: \"a189dbde-a4e1-40f6-90a5-745153598f1d\") "
Oct 30 22:23:21.482261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.482210    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-openshift-ca" pod "a189dbde-a4e1-40f6-90a5-745153598f1d" (UID: "a189dbde-a4e1-40f6-90a5-745153598f1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:21.482904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.482878    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:21.484103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.484081    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a189dbde-a4e1-40f6-90a5-745153598f1d-kube-api-access-7zz95" pod "a189dbde-a4e1-40f6-90a5-745153598f1d" (UID: "a189dbde-a4e1-40f6-90a5-745153598f1d"). InnerVolumeSpecName "kube-api-access-7zz95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:21.484254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.484133    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-xks-ca" pod "a189dbde-a4e1-40f6-90a5-745153598f1d" (UID: "a189dbde-a4e1-40f6-90a5-745153598f1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:21.584517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.584434    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a189dbde-a4e1-40f6-90a5-745153598f1d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:21.584517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.584473    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zz95\" (UniqueName: \"kubernetes.io/projected/a189dbde-a4e1-40f6-90a5-745153598f1d-kube-api-access-7zz95\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:21.617024 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.616988    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:21.623186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:21.623152    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:22.126426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.126387    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" event={"ID":"e09b6a1c-f6cb-45a6-a976-417dbbc90465","Type":"ContainerStarted","Data":"ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc"}
Oct 30 22:23:22.128268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.128245    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"
Oct 30 22:23:22.128268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.128255    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq" event={"ID":"a189dbde-a4e1-40f6-90a5-745153598f1d","Type":"ContainerDied","Data":"936796bdc68c6972195f8b0c3dba48e1510d325ef1c7bbdcc5043822b1cf520b"}
Oct 30 22:23:22.128469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.128295    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="936796bdc68c6972195f8b0c3dba48e1510d325ef1c7bbdcc5043822b1cf520b"
Oct 30 22:23:22.128469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.128399    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:22.129532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.129511    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:23:22.163532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.163481    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podStartSLOduration=2.163462625 podStartE2EDuration="2.163462625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.162367098 +0000 UTC m=+986.201305985" watchObservedRunningTime="2026-10-30 22:23:22.163462625 +0000 UTC m=+986.202401513"
Oct 30 22:23:22.647283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.647249    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:22.652851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:22.652823    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:23.157260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:23.157167    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:23.158531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:23.158505    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:23:27.083568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.083531    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"]
Oct 30 22:23:27.084394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.084376    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a189dbde-a4e1-40f6-90a5-745153598f1d" containerName="revoke-keys"
Oct 30 22:23:27.084453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.084396    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a189dbde-a4e1-40f6-90a5-745153598f1d" containerName="revoke-keys"
Oct 30 22:23:27.084577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.084563    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a189dbde-a4e1-40f6-90a5-745153598f1d" containerName="revoke-keys"
Oct 30 22:23:27.090501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.090470    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:27.108535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.108505    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"]
Oct 30 22:23:27.109309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.109282    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-36d7ee39\""
Oct 30 22:23:27.109437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.109336    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-36d7ee39-dockercfg-lb9v6\""
Oct 30 22:23:27.394309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.352050    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndng2\" (UniqueName: \"kubernetes.io/projected/56bd4f0f-ed17-4042-bad4-22075314e85c-kube-api-access-ndng2\") pod \"payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws\" (UID: \"56bd4f0f-ed17-4042-bad4-22075314e85c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:27.423248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.394606    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bd4f0f-ed17-4042-bad4-22075314e85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws\" (UID: \"56bd4f0f-ed17-4042-bad4-22075314e85c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:27.526234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.526184    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndng2\" (UniqueName: \"kubernetes.io/projected/56bd4f0f-ed17-4042-bad4-22075314e85c-kube-api-access-ndng2\") pod \"payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws\" (UID: \"56bd4f0f-ed17-4042-bad4-22075314e85c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:27.526437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.526277    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bd4f0f-ed17-4042-bad4-22075314e85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws\" (UID: \"56bd4f0f-ed17-4042-bad4-22075314e85c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:27.527201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.527166    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bd4f0f-ed17-4042-bad4-22075314e85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws\" (UID: \"56bd4f0f-ed17-4042-bad4-22075314e85c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:27.546434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.546400    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndng2\" (UniqueName: \"kubernetes.io/projected/56bd4f0f-ed17-4042-bad4-22075314e85c-kube-api-access-ndng2\") pod \"payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws\" (UID: \"56bd4f0f-ed17-4042-bad4-22075314e85c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:27.710675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:27.710574    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:28.148246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:28.143255    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"]
Oct 30 22:23:28.504682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:28.504642    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws" event={"ID":"56bd4f0f-ed17-4042-bad4-22075314e85c","Type":"ContainerStarted","Data":"5b63d640a7b5d342192123a29d4601f42dca40cce17d780acec23676c9f194e8"}
Oct 30 22:23:28.504864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:28.504700    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws" event={"ID":"56bd4f0f-ed17-4042-bad4-22075314e85c","Type":"ContainerStarted","Data":"68f5794288859838945393115546e9506fb70e1da7524dea856c4d8f5156fb69"}
Oct 30 22:23:28.504864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:28.504730    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:28.544958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:28.544898    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws" podStartSLOduration=1.544886017 podStartE2EDuration="1.544886017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:28.537090524 +0000 UTC m=+992.576029411" watchObservedRunningTime="2026-10-30 22:23:28.544886017 +0000 UTC m=+992.583824902"
Oct 30 22:23:33.788241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.779325    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"]
Oct 30 22:23:33.830294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.814441    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"]
Oct 30 22:23:33.830294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.814565    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:33.858295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.857580    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:23:33.858295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.857703    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:33.858295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.857801    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:33.858295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.857831    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:33.966410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.960125    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:33.966410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:33.960187    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrx9h\" (UniqueName: \"kubernetes.io/projected/5164dde6-b472-418d-98c3-51cd5b0009b1-kube-api-access-nrx9h\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.172250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.146728    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.217247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.216958    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.262526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.218113    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.262526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.251637    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.262526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.251704    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrx9h\" (UniqueName: \"kubernetes.io/projected/5164dde6-b472-418d-98c3-51cd5b0009b1-kube-api-access-nrx9h\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.262526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.251894    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.262526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.256135    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.431473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.368703    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.471245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.442069    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.628004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.471520    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.628586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.471560    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.640992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.640952    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrx9h\" (UniqueName: \"kubernetes.io/projected/5164dde6-b472-418d-98c3-51cd5b0009b1-kube-api-access-nrx9h\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.641268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.471625    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.657252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.605823    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334e36d8-r267h\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.657521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.624519    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.657851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.471593    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.805425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.793282    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": context deadline exceeded"
Oct 30 22:23:34.881894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.881631    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:34.970948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:34.970899    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.045713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.045670    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.245248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.241367    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.245248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.242183    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.245248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.243332    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:23:35.245248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.243390    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:23:35.455252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.452272    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.455252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.453904    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:23:35.589445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.589336    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.590147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.590120    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:23:35.634413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.634365    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.637254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.635406    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:23:35.641579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.641542    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.663449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.663406    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.690246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.667431    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:23:35.744485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.690608    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.745084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.690658    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.745603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.745582    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:23:35.746192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.746178    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:23:35.746422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:35.746398    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:23:36.030248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.005666    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:36.136370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.083606    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:23:36.181587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.165117    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:36.308384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.290109    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:23:36.308384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.299673    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.668253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.660756    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.768189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.768134    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:23:36.844504 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.844452    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.029307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.769042    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.057246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:37.039412    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.092259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:36.768579    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.187100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:37.184616    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:37.187100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:37.184703    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.187100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:37.184906    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.201209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:37.201165    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.290461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:37.290356    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.750249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:37.747258    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.750249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:37.306584    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.009791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:38.009309    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 22:23:38.300326 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:23:38.292608    4581 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.186s"
Oct 30 22:23:38.300326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:38.292708    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:23:39.132248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.127037    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:23:39.201051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.200816    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.206773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.206725    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.233414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.217282    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.234874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.217322    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.236101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.218547    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.236811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.218589    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.245363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.232956    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.280247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.232995    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.404482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.233038    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 30 22:23:39.404482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.233067    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.404482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.233091    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:39.404482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.289410    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:39.404482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.400762    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.404482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.331851    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:39.404482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.400826    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.405359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.329574    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 30 22:23:39.546263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.532027    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-29lrm" podUID="b7e5b659-2162-4552-919d-40c9866fd577" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.33:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.546263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.532177    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.657394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.655524    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:23:39.898372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.898322    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.924304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:39.901022    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.022611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.022565    4581 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:40.064884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.064836    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.187735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.187601    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.215340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.215298    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.234848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.234805    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.243810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.240352    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.247538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.240417    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.266501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.240441    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.288741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.288687    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.304934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.304873    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.422251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.415623    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.582026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.416839    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.582757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.416883    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:23:40.583325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.497596    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:40.583530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.583493    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:40.989246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:40.976821    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.242550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.242416    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:23:41.245210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.245177    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.253152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.253113    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.262380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.262339    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.262965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.262929    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.336500 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:23:41.336453    4581 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.17s"
Oct 30 22:23:41.336769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.336736    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h" event={"ID":"5164dde6-b472-418d-98c3-51cd5b0009b1","Type":"ContainerStarted","Data":"9c797fdbae6bff917a113716500d21a376b48a776b5c32f7e5f1e37a0eef3a98"}
Oct 30 22:23:41.337206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.337180    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:23:41.363001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.355920    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.363001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.356639    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.363001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.356944    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.426631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.423491    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:23:41.593631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.593514    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:23:41.612240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.594385    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:41.715874 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:23:41.715830    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:23:41.723174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.723143    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:23:41.723647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.723630    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:23:41.736049 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.736025    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"]
Oct 30 22:23:41.737990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.737968    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:23:41.746445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.738198    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:23:41.746445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.745706    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:23:41.770840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.768872    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:23:41.773546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.771929    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:23:41.796071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.787610    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:23:41.796071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.787655    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"]
Oct 30 22:23:41.823365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.815423    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h" event={"ID":"5164dde6-b472-418d-98c3-51cd5b0009b1","Type":"ContainerStarted","Data":"438956ccb3a63f989d5f2c8dc0314f1ded7e091e08fb29b525e65041fa1b964c"}
Oct 30 22:23:41.823365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.815574    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.839255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.833935    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"]
Oct 30 22:23:41.839255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.835367    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:23:41.863207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.850408    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"]
Oct 30 22:23:41.897514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.896945    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"]
Oct 30 22:23:41.897514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.897200    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:23:41.897514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.897400    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984119    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c720270-bcf8-433a-8edc-bb68f329640c-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.984348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984181    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984259    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984301    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.984348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984341    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984399    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984435    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5kx\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-kube-api-access-wr5kx\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.984563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984464    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c720270-bcf8-433a-8edc-bb68f329640c-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.984563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984498    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.984563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984550    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4485c1fc-ee74-4a23-a985-d74366141b6e-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984580    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984616    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.984778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984661    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984762    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvbc\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-kube-api-access-xvvbc\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:41.984958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.984920    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.985193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.985171    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.987738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.987706    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:41.987912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:41.987793    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.013999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.013935    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h" podStartSLOduration=9.013912781 podStartE2EDuration="9.013912781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:42.003510839 +0000 UTC m=+1006.042449726" watchObservedRunningTime="2026-10-30 22:23:42.013912781 +0000 UTC m=+1006.052851668"
Oct 30 22:23:42.105188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105088    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvbc\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-kube-api-access-xvvbc\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.105380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105272    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.105450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105438    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.105507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105480    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.105555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105528    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.105632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105613    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c720270-bcf8-433a-8edc-bb68f329640c-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.105682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105655    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.105737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105698    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.105831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105742    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.105881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105866    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.105941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105925    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.105992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.105974    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5kx\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-kube-api-access-wr5kx\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.106048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.106011    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c720270-bcf8-433a-8edc-bb68f329640c-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.106102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.106047    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.106157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.106101    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4485c1fc-ee74-4a23-a985-d74366141b6e-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.106157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.106144    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.106282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.106199    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.106429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.106271    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.110554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.110520    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c720270-bcf8-433a-8edc-bb68f329640c-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.110903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.110877    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.111534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.111505    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.111888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.111861    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.112959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.112932    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4485c1fc-ee74-4a23-a985-d74366141b6e-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.114875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.114847    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.116115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.116089    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.117466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.117442    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.117910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.117881    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.126574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.126536    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.127862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.127829    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.139867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.139836    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.158759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.158719    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.179435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.178039    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c720270-bcf8-433a-8edc-bb68f329640c-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.179435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.178536    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.182435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.182149    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5kx\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-kube-api-access-wr5kx\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.192151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.192047    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvbc\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-kube-api-access-xvvbc\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.192739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.192717    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.259896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.259638    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:42.275591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.275550    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:42.657210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.657127    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"]
Oct 30 22:23:42.670991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.670943    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:42.671114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.671040    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:42.671114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.671084    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:42.675170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.674330    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"]
Oct 30 22:23:42.689833 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.689793    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:42.689928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.689887    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:42.689982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.689933    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:42.829427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.828518    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" event={"ID":"8c720270-bcf8-433a-8edc-bb68f329640c","Type":"ContainerStarted","Data":"dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57"}
Oct 30 22:23:42.829427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.828583    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" event={"ID":"8c720270-bcf8-433a-8edc-bb68f329640c","Type":"ContainerStarted","Data":"b097a57db12ea77ca7436a1a371e092e3868de594e89a8d851b1be379022c2f3"}
Oct 30 22:23:42.837799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.837767    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" event={"ID":"4485c1fc-ee74-4a23-a985-d74366141b6e","Type":"ContainerStarted","Data":"bc66dad1c0a2c6c301e4b6d7d1814c7202ca7e66a9e62e912948bc909c155e89"}
Oct 30 22:23:42.838156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.838097    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" event={"ID":"4485c1fc-ee74-4a23-a985-d74366141b6e","Type":"ContainerStarted","Data":"8272f83056b1c0d36a9a4284ccc201c2256cc824d8f3d06c522b31b4244ce658"}
Oct 30 22:23:42.840436 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.840408    4581 generic.go:356] "Generic (PLEG): container finished" podID="5164dde6-b472-418d-98c3-51cd5b0009b1" containerID="438956ccb3a63f989d5f2c8dc0314f1ded7e091e08fb29b525e65041fa1b964c" exitCode=0
Oct 30 22:23:42.840680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.840661    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h" event={"ID":"5164dde6-b472-418d-98c3-51cd5b0009b1","Type":"ContainerDied","Data":"438956ccb3a63f989d5f2c8dc0314f1ded7e091e08fb29b525e65041fa1b964c"}
Oct 30 22:23:42.894281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.893806    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podStartSLOduration=4.893786724 podStartE2EDuration="4.893786724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:42.891556232 +0000 UTC m=+1006.930495121" watchObservedRunningTime="2026-10-30 22:23:42.893786724 +0000 UTC m=+1006.932725610"
Oct 30 22:23:42.942807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:42.939124    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podStartSLOduration=4.939102273 podStartE2EDuration="4.939102273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:42.928930103 +0000 UTC m=+1006.967868992" watchObservedRunningTime="2026-10-30 22:23:42.939102273 +0000 UTC m=+1006.978041160"
Oct 30 22:23:43.260607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:43.260516    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:43.262268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:43.262193    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:23:43.262402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:43.262291    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:23:43.281013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:43.278050    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:43.281013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:43.279824    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:43.281013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:43.279876    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:23:44.327043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.324337    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:23:44.327043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.326614    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:23:44.327043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.325790    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:44.327043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.326772    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:23:44.338655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.338247    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:44.539050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.538952    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-openshift-ca\") pod \"5164dde6-b472-418d-98c3-51cd5b0009b1\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") "
Oct 30 22:23:44.539459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.539435    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5164dde6-b472-418d-98c3-51cd5b0009b1-kube-api-access-nrx9h\" (UniqueName: \"kubernetes.io/projected/5164dde6-b472-418d-98c3-51cd5b0009b1-kube-api-access-nrx9h\") pod \"5164dde6-b472-418d-98c3-51cd5b0009b1\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") "
Oct 30 22:23:44.539643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.539626    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-xks-ca\") pod \"5164dde6-b472-418d-98c3-51cd5b0009b1\" (UID: \"5164dde6-b472-418d-98c3-51cd5b0009b1\") "
Oct 30 22:23:44.540080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.540049    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-openshift-ca" pod "5164dde6-b472-418d-98c3-51cd5b0009b1" (UID: "5164dde6-b472-418d-98c3-51cd5b0009b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:44.540598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.540568    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:44.544791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.544756    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5164dde6-b472-418d-98c3-51cd5b0009b1-kube-api-access-nrx9h" pod "5164dde6-b472-418d-98c3-51cd5b0009b1" (UID: "5164dde6-b472-418d-98c3-51cd5b0009b1"). InnerVolumeSpecName "kube-api-access-nrx9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:44.544879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.544805    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-xks-ca" pod "5164dde6-b472-418d-98c3-51cd5b0009b1" (UID: "5164dde6-b472-418d-98c3-51cd5b0009b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:44.641762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.641723    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrx9h\" (UniqueName: \"kubernetes.io/projected/5164dde6-b472-418d-98c3-51cd5b0009b1-kube-api-access-nrx9h\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:44.641762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.641754    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5164dde6-b472-418d-98c3-51cd5b0009b1-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:23:44.944577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.944517    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h" event={"ID":"5164dde6-b472-418d-98c3-51cd5b0009b1","Type":"ContainerDied","Data":"9c797fdbae6bff917a113716500d21a376b48a776b5c32f7e5f1e37a0eef3a98"}
Oct 30 22:23:44.944787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:44.944641    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c797fdbae6bff917a113716500d21a376b48a776b5c32f7e5f1e37a0eef3a98"
Oct 30 22:23:45.042245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:45.041278    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"
Oct 30 22:23:45.282362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:45.282323    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:45.282662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:45.282634    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:23:45.283441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:45.283416    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:23:45.283609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:45.283589    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:23:46.268356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:46.267359    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:23:46.268356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:46.267436    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:23:46.278832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:46.278674    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:46.278832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:46.278751    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:23:46.949249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:46.949186    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"]
Oct 30 22:23:46.989242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:46.981575    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5164dde6-b472-418d-98c3-51cd5b0009b1" containerName="revoke-keys"
Oct 30 22:23:46.989242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:46.981608    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5164dde6-b472-418d-98c3-51cd5b0009b1" containerName="revoke-keys"
Oct 30 22:23:46.989242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:46.981890    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5164dde6-b472-418d-98c3-51cd5b0009b1" containerName="revoke-keys"
Oct 30 22:23:47.240351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.219286    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.312243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.309424    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ggg7r\""
Oct 30 22:23:47.312243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.309501    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 22:23:47.312243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.309547    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 22:23:47.312243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.309589    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 22:23:47.312243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.310353    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:47.312243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.310409    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.359163    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhw7v\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-kube-api-access-xhw7v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.359352    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.359576    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.359624    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.359660    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.359719    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53888a7b-77ff-40c5-93ec-118de8225ffe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.359760    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.371588    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:23:47.372326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.371643    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:23:47.496680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.458322    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"]
Oct 30 22:23:47.496680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.468819    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.496680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.469017    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.496680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.469068    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.496680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.469105    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.496680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.469160    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53888a7b-77ff-40c5-93ec-118de8225ffe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.496680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.469203    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.496680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.469447    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhw7v\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-kube-api-access-xhw7v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.536038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.528210    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.541751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.540023    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.622365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.620737    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.622365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.621743    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53888a7b-77ff-40c5-93ec-118de8225ffe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.662246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.659059    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhw7v\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-kube-api-access-xhw7v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.662246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.659524    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.692266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.691976    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"]
Oct 30 22:23:47.715461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.715430    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47qt2\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.813342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.785209    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:47.850330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:47.850295    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcbfd6ff-zrvt7"]
Oct 30 22:23:48.293247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:48.278810    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:23:48.293247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:48.278886    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:23:48.293247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:48.280056    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:48.293247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:48.280096    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:23:48.533657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:48.533627    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78f705f4-0efd-4492-a198-c866ee875422" path="/var/lib/kubelet/pods/78f705f4-0efd-4492-a198-c866ee875422/volumes"
Oct 30 22:23:48.628005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:48.627772    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"]
Oct 30 22:23:48.723686 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:23:48.707376    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53888a7b_77ff_40c5_93ec_118de8225ffe.slice/crio-ac5c01e5156d494965662e19c70ddc3fbe5a482ba1eb701cf3fb3cd6cf441485 WatchSource:0}: Error finding container ac5c01e5156d494965662e19c70ddc3fbe5a482ba1eb701cf3fb3cd6cf441485: Status 404 returned error can't find the container with id ac5c01e5156d494965662e19c70ddc3fbe5a482ba1eb701cf3fb3cd6cf441485
Oct 30 22:23:48.774046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:48.771635    4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:23:49.277243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:49.277145    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:23:49.277243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:49.277206    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:23:49.283549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:49.278461    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:49.283549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:49.278527    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:23:49.631042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:49.631002    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2" event={"ID":"53888a7b-77ff-40c5-93ec-118de8225ffe","Type":"ContainerStarted","Data":"ac5c01e5156d494965662e19c70ddc3fbe5a482ba1eb701cf3fb3cd6cf441485"}
Oct 30 22:23:50.261669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:50.261160    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:23:50.261669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:50.261267    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:23:50.277818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:50.277783    4581 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:50.277990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:50.277844    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:23:51.462635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:51.462601    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:51.464541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:51.464512    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:51.466036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:51.466011    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:23:51.747115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:51.746994    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:51.748940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:51.748915    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:51.750179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:51.750154    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:23:52.868165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:52.868124    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:52.868564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:52.868210    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:23:53.694296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:53.694150    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2" event={"ID":"53888a7b-77ff-40c5-93ec-118de8225ffe","Type":"ContainerStarted","Data":"b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857"}
Oct 30 22:23:53.694296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:53.694280    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:23:53.699740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:53.699687    4581 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-47qt2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:23:53.699918 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:53.699809    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2" podUID="53888a7b-77ff-40c5-93ec-118de8225ffe" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:53.758854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:53.758794    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2" podStartSLOduration=3.664153916 podStartE2EDuration="7.758778276s" totalImagesPullingTime="4.09462436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:53.743568459 +0000 UTC m=+1017.782507346" watchObservedRunningTime="2026-10-30 22:23:53.758778276 +0000 UTC m=+1017.797717152"
Oct 30 22:23:54.704863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:23:54.704831    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:24:02.034314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:02.034274    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"]
Oct 30 22:24:02.042909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:02.042876    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786bd150-xz9vj"]
Oct 30 22:24:02.100406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:02.100369    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8113895f-1e4e-4742-84b7-2114231170bd" path="/var/lib/kubelet/pods/8113895f-1e4e-4742-84b7-2114231170bd/volumes"
Oct 30 22:24:03.032698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:03.032659    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"]
Oct 30 22:24:03.040581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:03.040545    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd8311b-fxzht"]
Oct 30 22:24:04.100278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:04.100243    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9562d9c5-c3b4-4ccf-906a-f1be94ba501e" path="/var/lib/kubelet/pods/9562d9c5-c3b4-4ccf-906a-f1be94ba501e/volumes"
Oct 30 22:24:09.248438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.248330    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.249289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.248339    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.295242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.294648    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.655245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.649413    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.665249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.662097    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"]
Oct 30 22:24:09.682244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.681430    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.807244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.802633    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.878438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.870167    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:24:09.887615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.887587    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:09.975987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.971021    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:09.975987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.971784    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:24:09.975987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.971854    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:09.975987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:09.971896    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:10.055325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.028571    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.055325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.044451    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.249281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.249156    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.277569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.277526    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:24:10.294252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.292987    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:24:10.299286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.299237    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:24:10.300969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.300943    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:24:10.375263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.362899    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjzt\" (UniqueName: \"kubernetes.io/projected/8eaba93e-51c7-4256-9061-430629dc59d3-kube-api-access-zpjzt\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.475437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.475393    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.500500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.500405    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.501402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.470472    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:24:10.502691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.471095    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.529584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.529546    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"]
Oct 30 22:24:10.629270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.624678    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjzt\" (UniqueName: \"kubernetes.io/projected/8eaba93e-51c7-4256-9061-430629dc59d3-kube-api-access-zpjzt\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.629270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.624757    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.629270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.624811    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.667975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.667519    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.669250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.668536    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.695244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.691344    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:24:10.777894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.720769    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:24:10.836258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.751069    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjzt\" (UniqueName: \"kubernetes.io/projected/8eaba93e-51c7-4256-9061-430629dc59d3-kube-api-access-zpjzt\") pod \"maas-api-revoke-keys-e2e-ait-ca259734-4sq4r\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.918079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.870055    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:10.918079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.875747    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.918079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.875834    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:24:10.918079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.907738    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:24:10.918079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:10.914612    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.086346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.070700    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.087952 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.087920    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:24:11.089390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.089356    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.089813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.089792    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:24:11.274426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.274280    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.317250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.301183    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.317250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.305266    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.317250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.314859    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:24:11.443527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.329884    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:24:11.750452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.661257    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.750452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.690397    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:24:11.750452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.710853    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:24:11.778245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.752531    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:24:11.778245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.753875    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:24:11.876798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.876741    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:24:11.882532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.882483    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded"
Oct 30 22:24:11.908292 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:24:11.900424    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:24:11.917774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.916593    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.974358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:11.942664    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:24:12.133431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.133388    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": context deadline exceeded"
Oct 30 22:24:12.412245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.410618    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4" podUID="17847819-d7b1-4946-8a3c-326d50cdd08e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.745543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.572465    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.745543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.683633    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.745543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.711401    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.817449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.683582    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.818132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.818037    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:24:12.819414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.819382    4581 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-plmfm container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.133.0.14:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:24:12.819657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.819615    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-plmfm" podUID="3b303815-9277-4c80-9f8b-14fed36152b3" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.877427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.877319    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:12.942615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:12.942509    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.072209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.072067    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.387351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.387308    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xdzkp" podUID="90d4ed54-8bcf-442b-82c4-cbae47556237" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 22:24:13.388098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.388068    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:13.447069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.447028    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.658858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.658756    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.690401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.690356    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.831394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.831346    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:24:13.833008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.832974    4581 patch_prober.go:29] interesting pod/jobset-controller-manager-6c87f6c66-mlkmg container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:24:13.833243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.833198    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-6c87f6c66-mlkmg" podUID="bbd23474-9030-4f92-9e06-35ba36ab122e" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:13.858999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.858956    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:13.956310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:13.924187    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.121245 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:24:14.089462    4581 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.89s"
Oct 30 22:24:14.121245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.089572    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:24:14.277306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.260683    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:24:14.395956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.395909    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.454462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.454403    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:24:14.693095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.693041    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.702922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.676363    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.847106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.847047    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.907408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.907366    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:24:14.908262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.908228    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:24:14.911294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.911265    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.915973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.915018    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:14.932241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.928309    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.275956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.261355    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.310785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:14.878332    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.396684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.396627    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:24:15.538529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.538425    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:24:15.706245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.694482    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.718249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.716090    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:24:15.875310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.864032    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.908288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.908236    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:15.977254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.967870    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:24:16.157265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.908552    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:16.157265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:15.991835    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:24:16.208457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.179291    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sv4fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:24:16.208457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.179355    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:16.272930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.272880    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:16.388302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.376477    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:16.437767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.425409    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:16.697565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.697454    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:24:16.838496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.770640    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:24:16.924614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.924564    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:17.011309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:16.941158    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:24:17.091278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.089247    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:24:17.218956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.218856    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:17.268250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.240460    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:24:17.268250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.242967    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:24:17.504136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.504027    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r" event={"ID":"8eaba93e-51c7-4256-9061-430629dc59d3","Type":"ContainerStarted","Data":"f891017470dbdd804a6be2d75ee3cbd047c3d2ab177e29575d98dd0f80bc3f80"}
Oct 30 22:24:17.593654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.593617    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:24:17.784403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.781685    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:17.799442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.799397    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:24:17.800059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.799706    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:17.816073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.815082    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:17.816073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.815147    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:24:17.844782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.844733    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:17.870818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.867034    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:17.872167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.872135    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vbv8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:24:17.872408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.872373    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:17.989203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:17.988306    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:24:18.013392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.013345    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:24:18.101537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.101478    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:18.251929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.251876    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:18.260081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.254157    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:18.490242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.489588    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"]
Oct 30 22:24:18.636285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.633908    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"]
Oct 30 22:24:18.654111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.654080    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:18.655385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.653986    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"]
Oct 30 22:24:18.733450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.733411    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"]
Oct 30 22:24:18.787482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.782206    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:18.819571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.802353    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:24:18.819571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.802633    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:24:18.819571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.815471    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:18.831592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.831552    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:18.831935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.831907    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpq5\" (UniqueName: \"kubernetes.io/projected/c298b3b2-cbab-4565-a7a3-93509aca60cc-kube-api-access-kcpq5\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:18.832102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.832083    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:18.850084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.837110    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r" event={"ID":"8eaba93e-51c7-4256-9061-430629dc59d3","Type":"ContainerStarted","Data":"6916c8709c50fce9a6b94200ac688d7d2804a153b07bc3545b0719f89b729cce"}
Oct 30 22:24:18.851520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.850416    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:24:18.905966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.905933    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86710a-nbwlh"]
Oct 30 22:24:18.954252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.951665    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:18.954252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.951805    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpq5\" (UniqueName: \"kubernetes.io/projected/c298b3b2-cbab-4565-a7a3-93509aca60cc-kube-api-access-kcpq5\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:18.954252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.951853    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:18.982247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.956644    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.006245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:18.990576    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.009026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.008991    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:19.017363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.017310    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.047305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.043335    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.086303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.085256    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpq5\" (UniqueName: \"kubernetes.io/projected/c298b3b2-cbab-4565-a7a3-93509aca60cc-kube-api-access-kcpq5\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:19.086303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.085602    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:19.088021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.086770    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.107812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.107769    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.201845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.201809    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:24:19.228268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.206264    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"]
Oct 30 22:24:19.245190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.245159    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"]
Oct 30 22:24:19.270475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.270431    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.271104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.271072    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:24:19.389234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.388956    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:19.471494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.471408    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716b7868-m22hj"]
Oct 30 22:24:19.515353 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.506288    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c82a37-lxmqh"]
Oct 30 22:24:19.595786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.547866    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:24:19.636852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.636789    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r" podStartSLOduration=10.636766135 podStartE2EDuration="10.636766135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:18.872959433 +0000 UTC m=+1042.911898319" watchObservedRunningTime="2026-10-30 22:24:19.636766135 +0000 UTC m=+1043.675705023"
Oct 30 22:24:19.641456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.641281    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 22:24:19.649266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.649243    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:24:19.823873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.823819    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:24:19.888145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.851598    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.948842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.948734    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:19.990272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.989298    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:24:19.996251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:19.990707    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:24:20.019636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.019572    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:20.049282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.048421    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:24:20.127445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.115871    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:24:20.180745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.180697    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:20.293367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.275476    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:24:20.497426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.497389    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a2e504-9393-4da1-bd67-4b58fa4510a2" path="/var/lib/kubelet/pods/a9a2e504-9393-4da1-bd67-4b58fa4510a2/volumes"
Oct 30 22:24:20.514445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.498578    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74c0295-9424-462a-95d9-7170777ed5e2" path="/var/lib/kubelet/pods/d74c0295-9424-462a-95d9-7170777ed5e2/volumes"
Oct 30 22:24:20.554548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.554434    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:24:20.729813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.729774    4581 generic.go:356] "Generic (PLEG): container finished" podID="8eaba93e-51c7-4256-9061-430629dc59d3" containerID="6916c8709c50fce9a6b94200ac688d7d2804a153b07bc3545b0719f89b729cce" exitCode=0
Oct 30 22:24:20.744869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.744835    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafbf2ee-8c6d-4897-a701-6f90ad2f3422" path="/var/lib/kubelet/pods/dafbf2ee-8c6d-4897-a701-6f90ad2f3422/volumes"
Oct 30 22:24:20.745633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.745613    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:24:20.745810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.745800    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:24:20.745934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.745925    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:24:20.746027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.746010    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r" event={"ID":"8eaba93e-51c7-4256-9061-430629dc59d3","Type":"ContainerDied","Data":"6916c8709c50fce9a6b94200ac688d7d2804a153b07bc3545b0719f89b729cce"}
Oct 30 22:24:20.746150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.746138    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:24:20.746278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.746268    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:24:20.746379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.746370    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:24:20.746486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.746478    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:24:20.942258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:20.938284    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"]
Oct 30 22:24:21.198472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.198334    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df6b8d3-45mqq"]
Oct 30 22:24:21.235831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.235788    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"]
Oct 30 22:24:21.244521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.243339    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.251920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.251829    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"]
Oct 30 22:24:21.344846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.344265    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.344846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.344345    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.344846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.344445    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pll68\" (UniqueName: \"kubernetes.io/projected/a39f7f88-4626-46c4-8b06-e1675ba04a54-kube-api-access-pll68\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.432542 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:24:21.432423    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc298b3b2_cbab_4565_a7a3_93509aca60cc.slice/crio-829df1969dd401218a020d05cafe2982312c2173b6cf853230f6db7facad399e WatchSource:0}: Error finding container 829df1969dd401218a020d05cafe2982312c2173b6cf853230f6db7facad399e: Status 404 returned error can't find the container with id 829df1969dd401218a020d05cafe2982312c2173b6cf853230f6db7facad399e
Oct 30 22:24:21.433873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.433744    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"]
Oct 30 22:24:21.446921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.446284    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.446921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.446346    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.446921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.446423    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pll68\" (UniqueName: \"kubernetes.io/projected/a39f7f88-4626-46c4-8b06-e1675ba04a54-kube-api-access-pll68\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.447883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.447857    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.450617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.450536    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.466818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.466781    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pll68\" (UniqueName: \"kubernetes.io/projected/a39f7f88-4626-46c4-8b06-e1675ba04a54-kube-api-access-pll68\") pod \"maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.613555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.613452    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:21.778812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.778769    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp" event={"ID":"c298b3b2-cbab-4565-a7a3-93509aca60cc","Type":"ContainerStarted","Data":"829df1969dd401218a020d05cafe2982312c2173b6cf853230f6db7facad399e"}
Oct 30 22:24:21.960926 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:24:21.960818    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f7f88_4626_46c4_8b06_e1675ba04a54.slice/crio-8eaf6ee38b3209cb0c893231d0adde828af141c0388dfcf8c60f7bdb77224b3c WatchSource:0}: Error finding container 8eaf6ee38b3209cb0c893231d0adde828af141c0388dfcf8c60f7bdb77224b3c: Status 404 returned error can't find the container with id 8eaf6ee38b3209cb0c893231d0adde828af141c0388dfcf8c60f7bdb77224b3c
Oct 30 22:24:21.963893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:21.963340    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"]
Oct 30 22:24:22.134511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.134471    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c6a047-d46c-43cb-88b1-7b9f9b6576db" path="/var/lib/kubelet/pods/a7c6a047-d46c-43cb-88b1-7b9f9b6576db/volumes"
Oct 30 22:24:22.148478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.148441    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:22.269928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.269892    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-openshift-ca\") pod \"8eaba93e-51c7-4256-9061-430629dc59d3\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") "
Oct 30 22:24:22.270106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.269998    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eaba93e-51c7-4256-9061-430629dc59d3-kube-api-access-zpjzt\" (UniqueName: \"kubernetes.io/projected/8eaba93e-51c7-4256-9061-430629dc59d3-kube-api-access-zpjzt\") pod \"8eaba93e-51c7-4256-9061-430629dc59d3\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") "
Oct 30 22:24:22.270167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.270120    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-xks-ca\") pod \"8eaba93e-51c7-4256-9061-430629dc59d3\" (UID: \"8eaba93e-51c7-4256-9061-430629dc59d3\") "
Oct 30 22:24:22.270666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.270635    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-openshift-ca" pod "8eaba93e-51c7-4256-9061-430629dc59d3" (UID: "8eaba93e-51c7-4256-9061-430629dc59d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:22.273772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.273743    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eaba93e-51c7-4256-9061-430629dc59d3-kube-api-access-zpjzt" pod "8eaba93e-51c7-4256-9061-430629dc59d3" (UID: "8eaba93e-51c7-4256-9061-430629dc59d3"). InnerVolumeSpecName "kube-api-access-zpjzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:22.273891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.273866    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-xks-ca" pod "8eaba93e-51c7-4256-9061-430629dc59d3" (UID: "8eaba93e-51c7-4256-9061-430629dc59d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:22.373546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.373201    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjzt\" (UniqueName: \"kubernetes.io/projected/8eaba93e-51c7-4256-9061-430629dc59d3-kube-api-access-zpjzt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:22.373761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.373740    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:22.373916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.373898    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaba93e-51c7-4256-9061-430629dc59d3-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:22.797822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.797777    4581 generic.go:356] "Generic (PLEG): container finished" podID="c298b3b2-cbab-4565-a7a3-93509aca60cc" containerID="97ab4d1c8410ab3036e2712361c2ec0bc30b1f1ef741fa39de46c3a358147333" exitCode=0
Oct 30 22:24:22.798021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.797904    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp" event={"ID":"c298b3b2-cbab-4565-a7a3-93509aca60cc","Type":"ContainerDied","Data":"97ab4d1c8410ab3036e2712361c2ec0bc30b1f1ef741fa39de46c3a358147333"}
Oct 30 22:24:22.800317 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.800288    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"
Oct 30 22:24:22.804238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.800494    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r" event={"ID":"8eaba93e-51c7-4256-9061-430629dc59d3","Type":"ContainerDied","Data":"f891017470dbdd804a6be2d75ee3cbd047c3d2ab177e29575d98dd0f80bc3f80"}
Oct 30 22:24:22.804238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.800546    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f891017470dbdd804a6be2d75ee3cbd047c3d2ab177e29575d98dd0f80bc3f80"
Oct 30 22:24:22.812597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.812188    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j" event={"ID":"a39f7f88-4626-46c4-8b06-e1675ba04a54","Type":"ContainerStarted","Data":"d72569e505cde1ea477941ab28b929234ef06234e2bc16fdf8d33caf747bdbf7"}
Oct 30 22:24:22.812597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:22.812260    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j" event={"ID":"a39f7f88-4626-46c4-8b06-e1675ba04a54","Type":"ContainerStarted","Data":"8eaf6ee38b3209cb0c893231d0adde828af141c0388dfcf8c60f7bdb77224b3c"}
Oct 30 22:24:23.179060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:23.178701    4581 patch_prober.go:29] interesting pod/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:24:23.179060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:23.178765    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:23.218813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:23.218738    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j" podStartSLOduration=3.218716707 podStartE2EDuration="3.218716707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:22.882281367 +0000 UTC m=+1046.921220254" watchObservedRunningTime="2026-10-30 22:24:23.218716707 +0000 UTC m=+1047.257655594"
Oct 30 22:24:23.829660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:23.829618    4581 generic.go:356] "Generic (PLEG): container finished" podID="a39f7f88-4626-46c4-8b06-e1675ba04a54" containerID="d72569e505cde1ea477941ab28b929234ef06234e2bc16fdf8d33caf747bdbf7" exitCode=0
Oct 30 22:24:23.830731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:23.830684    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j" event={"ID":"a39f7f88-4626-46c4-8b06-e1675ba04a54","Type":"ContainerDied","Data":"d72569e505cde1ea477941ab28b929234ef06234e2bc16fdf8d33caf747bdbf7"}
Oct 30 22:24:24.198535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.198418    4581 patch_prober.go:29] interesting pod/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:24:24.199873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.198516    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:24.208817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.208785    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:24.343059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.343013    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-xks-ca\") pod \"c298b3b2-cbab-4565-a7a3-93509aca60cc\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") "
Oct 30 22:24:24.343296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.343238    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c298b3b2-cbab-4565-a7a3-93509aca60cc-kube-api-access-kcpq5\" (UniqueName: \"kubernetes.io/projected/c298b3b2-cbab-4565-a7a3-93509aca60cc-kube-api-access-kcpq5\") pod \"c298b3b2-cbab-4565-a7a3-93509aca60cc\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") "
Oct 30 22:24:24.343393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.343374    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-openshift-ca\") pod \"c298b3b2-cbab-4565-a7a3-93509aca60cc\" (UID: \"c298b3b2-cbab-4565-a7a3-93509aca60cc\") "
Oct 30 22:24:24.344884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.344852    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-openshift-ca" pod "c298b3b2-cbab-4565-a7a3-93509aca60cc" (UID: "c298b3b2-cbab-4565-a7a3-93509aca60cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:24.364893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.364818    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c298b3b2-cbab-4565-a7a3-93509aca60cc-kube-api-access-kcpq5" pod "c298b3b2-cbab-4565-a7a3-93509aca60cc" (UID: "c298b3b2-cbab-4565-a7a3-93509aca60cc"). InnerVolumeSpecName "kube-api-access-kcpq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:24.364977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.364950    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-xks-ca" pod "c298b3b2-cbab-4565-a7a3-93509aca60cc" (UID: "c298b3b2-cbab-4565-a7a3-93509aca60cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:24.445586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.445550    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:24.445586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.445579    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpq5\" (UniqueName: \"kubernetes.io/projected/c298b3b2-cbab-4565-a7a3-93509aca60cc-kube-api-access-kcpq5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:24.445586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.445589    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c298b3b2-cbab-4565-a7a3-93509aca60cc-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:24.845928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.845821    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"
Oct 30 22:24:24.846116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.845820    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp" event={"ID":"c298b3b2-cbab-4565-a7a3-93509aca60cc","Type":"ContainerDied","Data":"829df1969dd401218a020d05cafe2982312c2173b6cf853230f6db7facad399e"}
Oct 30 22:24:24.846116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:24.845982    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="829df1969dd401218a020d05cafe2982312c2173b6cf853230f6db7facad399e"
Oct 30 22:24:25.149112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.149077    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:25.310367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.303333    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-openshift-ca\") pod \"a39f7f88-4626-46c4-8b06-e1675ba04a54\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") "
Oct 30 22:24:25.310367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.303401    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-xks-ca\") pod \"a39f7f88-4626-46c4-8b06-e1675ba04a54\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") "
Oct 30 22:24:25.310367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.303435    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39f7f88-4626-46c4-8b06-e1675ba04a54-kube-api-access-pll68\" (UniqueName: \"kubernetes.io/projected/a39f7f88-4626-46c4-8b06-e1675ba04a54-kube-api-access-pll68\") pod \"a39f7f88-4626-46c4-8b06-e1675ba04a54\" (UID: \"a39f7f88-4626-46c4-8b06-e1675ba04a54\") "
Oct 30 22:24:25.314849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.314804    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-openshift-ca" pod "a39f7f88-4626-46c4-8b06-e1675ba04a54" (UID: "a39f7f88-4626-46c4-8b06-e1675ba04a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:25.315389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.315362    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:25.320267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.320202    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-xks-ca" pod "a39f7f88-4626-46c4-8b06-e1675ba04a54" (UID: "a39f7f88-4626-46c4-8b06-e1675ba04a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:25.322733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.322676    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f7f88-4626-46c4-8b06-e1675ba04a54-kube-api-access-pll68" pod "a39f7f88-4626-46c4-8b06-e1675ba04a54" (UID: "a39f7f88-4626-46c4-8b06-e1675ba04a54"). InnerVolumeSpecName "kube-api-access-pll68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:25.417065 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.416970    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39f7f88-4626-46c4-8b06-e1675ba04a54-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:25.417065 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.417015    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pll68\" (UniqueName: \"kubernetes.io/projected/a39f7f88-4626-46c4-8b06-e1675ba04a54-kube-api-access-pll68\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:25.870807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.870329    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j" event={"ID":"a39f7f88-4626-46c4-8b06-e1675ba04a54","Type":"ContainerDied","Data":"8eaf6ee38b3209cb0c893231d0adde828af141c0388dfcf8c60f7bdb77224b3c"}
Oct 30 22:24:25.870807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.870447    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eaf6ee38b3209cb0c893231d0adde828af141c0388dfcf8c60f7bdb77224b3c"
Oct 30 22:24:25.870807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:25.870567    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"
Oct 30 22:24:26.973231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:26.887880    4581 generic.go:356] "Generic (PLEG): container finished" podID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerID="ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc" exitCode=0
Oct 30 22:24:26.973231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:26.887924    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" event={"ID":"e09b6a1c-f6cb-45a6-a976-417dbbc90465","Type":"ContainerDied","Data":"ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc"}
Oct 30 22:24:26.973231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:26.888495    4581 scope.go:122] "RemoveContainer" containerID="ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc"
Oct 30 22:24:26.973231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:26.890356    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:24:26.973231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:26.890434    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:24:26.973231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:26.890470    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:24:27.928255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:27.925656    4581 generic.go:356] "Generic (PLEG): container finished" podID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerID="6970c30ee638e8a4968447fab5ce43a2b48c01a4777a649005b3cb6e46b730eb" exitCode=0
Oct 30 22:24:27.928255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:27.925731    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" event={"ID":"8880fff5-a9b3-4522-a4cf-8bba135c5d99","Type":"ContainerDied","Data":"6970c30ee638e8a4968447fab5ce43a2b48c01a4777a649005b3cb6e46b730eb"}
Oct 30 22:24:27.928255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:27.927671    4581 scope.go:122] "RemoveContainer" containerID="6970c30ee638e8a4968447fab5ce43a2b48c01a4777a649005b3cb6e46b730eb"
Oct 30 22:24:27.929548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:27.928575    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" event={"ID":"e09b6a1c-f6cb-45a6-a976-417dbbc90465","Type":"ContainerStarted","Data":"450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166"}
Oct 30 22:24:27.930492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:27.930458    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:24:27.930611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:27.930552    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:24:27.930611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:27.930585    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:24:28.646529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:28.646499    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:24:28.647037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:28.646667    4581 patch_prober.go:29] interesting pod/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:24:28.647037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:28.646732    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:24:28.941455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:28.941357    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" event={"ID":"8880fff5-a9b3-4522-a4cf-8bba135c5d99","Type":"ContainerStarted","Data":"6550d9167fac69552228ac39cc1a412b90fe6d0943f82c236dec8de484fb74aa"}
Oct 30 22:24:29.036597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.036558    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"]
Oct 30 22:24:29.043854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.043825    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"]
Oct 30 22:24:29.073245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.067616    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05cea822-vqc78"]
Oct 30 22:24:29.113245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.105340    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439b6d2-rv2k4"]
Oct 30 22:24:29.617609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.617569    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:24:29.618104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.618075    4581 patch_prober.go:29] interesting pod/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 22:24:29.618243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.618128    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 22:24:29.647042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.647007    4581 patch_prober.go:29] interesting pod/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:24:29.647474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:29.647059    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:24:30.125135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:30.125096    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce5b204-69a6-4e9e-b574-4aac213801ad" path="/var/lib/kubelet/pods/3ce5b204-69a6-4e9e-b574-4aac213801ad/volumes"
Oct 30 22:24:30.125703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:30.125686    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87e0e75-f6eb-46fa-893f-c59a07514953" path="/var/lib/kubelet/pods/b87e0e75-f6eb-46fa-893f-c59a07514953/volumes"
Oct 30 22:24:30.628118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:30.623652    4581 patch_prober.go:29] interesting pod/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 22:24:30.628118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:30.623714    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 22:24:30.669139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:30.668983    4581 patch_prober.go:29] interesting pod/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:24:30.669139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:30.669045    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:24:31.109392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:31.109363    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"]
Oct 30 22:24:31.157243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:31.153704    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba606e-9php9"]
Oct 30 22:24:31.239988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:31.235856    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"]
Oct 30 22:24:31.625242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:31.621616    4581 patch_prober.go:29] interesting pod/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 22:24:31.625242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:31.621712    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 22:24:31.719933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:31.719453    4581 patch_prober.go:29] interesting pod/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:24:31.719933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:31.719512    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:24:32.333360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:32.333313    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6" path="/var/lib/kubelet/pods/5e5ac55e-fbc2-4a53-b3ac-1911b9e28ae6/volumes"
Oct 30 22:24:32.617519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:32.617485    4581 patch_prober.go:29] interesting pod/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 22:24:32.617782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:32.617565    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 22:24:32.647358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:32.647326    4581 patch_prober.go:29] interesting pod/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:24:32.647514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:32.647392    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:24:33.035469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:33.035377    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"]
Oct 30 22:24:33.043270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:33.043233    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589cf440-md8w7"]
Oct 30 22:24:33.049906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:33.049881    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"]
Oct 30 22:24:33.094789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:33.094751    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8eac990-mbjjd"]
Oct 30 22:24:33.621203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:33.621167    4581 patch_prober.go:29] interesting pod/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 22:24:33.621534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:33.621504    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 22:24:33.659135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:33.659100    4581 patch_prober.go:29] interesting pod/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:24:33.659444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:33.659415    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:24:34.672270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.657445    4581 patch_prober.go:29] interesting pod/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 22:24:34.672270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.657535    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 22:24:34.674118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.673150    4581 patch_prober.go:29] interesting pod/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:24:34.674118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.673202    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:24:34.701820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.700445    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c38381f-e41e-44de-bbb0-19da746b43ca" path="/var/lib/kubelet/pods/9c38381f-e41e-44de-bbb0-19da746b43ca/volumes"
Oct 30 22:24:34.701820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.701128    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81ebe40-59bb-4bb4-9d9a-c94698a04efa" path="/var/lib/kubelet/pods/b81ebe40-59bb-4bb4-9d9a-c94698a04efa/volumes"
Oct 30 22:24:34.701820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.701648    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"]
Oct 30 22:24:34.701820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.701689    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac8f84-cdzjh"]
Oct 30 22:24:34.747275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.746347    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"]
Oct 30 22:24:34.757374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.757328    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:34.821382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.818311    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca259734-4sq4r"]
Oct 30 22:24:34.821382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.818417    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:34.879387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.879333    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:34.888252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.887641    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"]
Oct 30 22:24:34.888252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:34.887896    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy" containerID="cri-o://6550d9167fac69552228ac39cc1a412b90fe6d0943f82c236dec8de484fb74aa" gracePeriod=30
Oct 30 22:24:35.192259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:35.181675    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:24:35.306583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:35.306518    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"]
Oct 30 22:24:35.318879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:35.318845    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ba9ae08-459xp"]
Oct 30 22:24:35.320622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:35.320558    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:24:35.545043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:35.544937    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:24:35.808323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:35.795454    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:35.808323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:35.796578    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:24:35.874090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:35.874057    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:24:36.009939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.009906    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:24:36.011299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.011275    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:24:36.012761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.012732    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:24:36.140540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.140504    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eaba93e-51c7-4256-9061-430629dc59d3" path="/var/lib/kubelet/pods/8eaba93e-51c7-4256-9061-430629dc59d3/volumes"
Oct 30 22:24:36.173329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.173292    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c298b3b2-cbab-4565-a7a3-93509aca60cc" path="/var/lib/kubelet/pods/c298b3b2-cbab-4565-a7a3-93509aca60cc/volumes"
Oct 30 22:24:36.174398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.174312    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d100333d-7032-45a7-abf3-108111c7ced1" path="/var/lib/kubelet/pods/d100333d-7032-45a7-abf3-108111c7ced1/volumes"
Oct 30 22:24:36.261326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.261274    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws" podUID="56bd4f0f-ed17-4042-bad4-22075314e85c" containerName="payload-pre-processing" containerID="cri-o://5b63d640a7b5d342192123a29d4601f42dca40cce17d780acec23676c9f194e8" gracePeriod=25
Oct 30 22:24:36.709013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.708984    4581 generic.go:356] "Generic (PLEG): container finished" podID="56bd4f0f-ed17-4042-bad4-22075314e85c" containerID="5b63d640a7b5d342192123a29d4601f42dca40cce17d780acec23676c9f194e8" exitCode=0
Oct 30 22:24:36.709161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.709060    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws" event={"ID":"56bd4f0f-ed17-4042-bad4-22075314e85c","Type":"ContainerDied","Data":"5b63d640a7b5d342192123a29d4601f42dca40cce17d780acec23676c9f194e8"}
Oct 30 22:24:36.709161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.709127    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws" event={"ID":"56bd4f0f-ed17-4042-bad4-22075314e85c","Type":"ContainerDied","Data":"68f5794288859838945393115546e9506fb70e1da7524dea856c4d8f5156fb69"}
Oct 30 22:24:36.709338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.709163    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68f5794288859838945393115546e9506fb70e1da7524dea856c4d8f5156fb69"
Oct 30 22:24:36.733132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.733103    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:24:36.794183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.794150    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"]
Oct 30 22:24:36.887650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.887603    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bd4f0f-ed17-4042-bad4-22075314e85c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bd4f0f-ed17-4042-bad4-22075314e85c-plugins-config-volume\") pod \"56bd4f0f-ed17-4042-bad4-22075314e85c\" (UID: \"56bd4f0f-ed17-4042-bad4-22075314e85c\") "
Oct 30 22:24:36.888072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.887769    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bd4f0f-ed17-4042-bad4-22075314e85c-kube-api-access-ndng2\" (UniqueName: \"kubernetes.io/projected/56bd4f0f-ed17-4042-bad4-22075314e85c-kube-api-access-ndng2\") pod \"56bd4f0f-ed17-4042-bad4-22075314e85c\" (UID: \"56bd4f0f-ed17-4042-bad4-22075314e85c\") "
Oct 30 22:24:36.888597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.888557    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bd4f0f-ed17-4042-bad4-22075314e85c-plugins-config-volume" pod "56bd4f0f-ed17-4042-bad4-22075314e85c" (UID: "56bd4f0f-ed17-4042-bad4-22075314e85c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:36.892114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:36.892064    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bd4f0f-ed17-4042-bad4-22075314e85c-kube-api-access-ndng2" pod "56bd4f0f-ed17-4042-bad4-22075314e85c" (UID: "56bd4f0f-ed17-4042-bad4-22075314e85c"). InnerVolumeSpecName "kube-api-access-ndng2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:37.043301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:37.009141    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bd4f0f-ed17-4042-bad4-22075314e85c-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:37.043301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:37.009187    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndng2\" (UniqueName: \"kubernetes.io/projected/56bd4f0f-ed17-4042-bad4-22075314e85c-kube-api-access-ndng2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:37.129049 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:37.129020    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:24:37.743863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:37.743830    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"
Oct 30 22:24:37.744318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:37.744286    4581 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" secret="" err="secret \"e2e-ait-1ba9ae08-openshift-default-dockercfg-nnpzt\" not found"
Oct 30 22:24:37.802255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:37.802209    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"]
Oct 30 22:24:37.838650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:37.838613    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-36d7ee39-69869b5bc-7jmws"]
Oct 30 22:24:38.148388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:38.147873    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bd4f0f-ed17-4042-bad4-22075314e85c" path="/var/lib/kubelet/pods/56bd4f0f-ed17-4042-bad4-22075314e85c/volumes"
Oct 30 22:24:38.489605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:38.489515    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"]
Oct 30 22:24:38.502163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:38.502116    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-36d7ee39-k6d4j"]
Oct 30 22:24:38.760883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:38.760760    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy" containerID="cri-o://450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166" gracePeriod=30
Oct 30 22:24:40.126240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:40.125280    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39f7f88-4626-46c4-8b06-e1675ba04a54" path="/var/lib/kubelet/pods/a39f7f88-4626-46c4-8b06-e1675ba04a54/volumes"
Oct 30 22:24:40.826029 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:24:40.825998    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:24:41.802187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:41.802153    4581 generic.go:356] "Generic (PLEG): container finished" podID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerID="6550d9167fac69552228ac39cc1a412b90fe6d0943f82c236dec8de484fb74aa" exitCode=0
Oct 30 22:24:41.802643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:41.802254    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" event={"ID":"8880fff5-a9b3-4522-a4cf-8bba135c5d99","Type":"ContainerDied","Data":"6550d9167fac69552228ac39cc1a412b90fe6d0943f82c236dec8de484fb74aa"}
Oct 30 22:24:41.802643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:41.802337    4581 scope.go:122] "RemoveContainer" containerID="6970c30ee638e8a4968447fab5ce43a2b48c01a4777a649005b3cb6e46b730eb"
Oct 30 22:24:42.042731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.042700    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:24:42.181689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.181644    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-podinfo\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.181881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.181741    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-certs\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.181881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.181813    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-envoy\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.181881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.181854    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-kube-api-access-rffxt\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-kube-api-access-rffxt\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.182065 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.181924    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-credential-socket\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.182065 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182038    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istiod-ca-cert\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.182177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182067    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-socket\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.182177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182085    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-certs" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:42.182177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182118    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-data\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.182940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182504    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istiod-ca-cert" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:42.182940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182497    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-credential-socket" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:42.182940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182516    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-socket" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:42.182940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182662    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-token\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-token\") pod \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\" (UID: \"8880fff5-a9b3-4522-a4cf-8bba135c5d99\") "
Oct 30 22:24:42.182940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.182874    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-data" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:42.183929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.183904    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.184015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.183932    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.184015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.183946    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.184015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.183960    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.184015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.183973    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.184442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.184417    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-envoy" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:42.185557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.185426    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-podinfo" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:42.185557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.185463    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-token" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:42.185557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.185520    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-kube-api-access-rffxt" pod "8880fff5-a9b3-4522-a4cf-8bba135c5d99" (UID: "8880fff5-a9b3-4522-a4cf-8bba135c5d99"). InnerVolumeSpecName "kube-api-access-rffxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:42.285697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.285605    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.285697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.285649    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rffxt\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-kube-api-access-rffxt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.285697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.285666    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.285697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.285678    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8880fff5-a9b3-4522-a4cf-8bba135c5d99-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:42.813380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.813338    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"
Oct 30 22:24:42.813380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.813378    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz" event={"ID":"8880fff5-a9b3-4522-a4cf-8bba135c5d99","Type":"ContainerDied","Data":"bcd80093dba468c4008362b8b035f774dde533dac3164df29485341d8c340758"}
Oct 30 22:24:42.813892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.813435    4581 scope.go:122] "RemoveContainer" containerID="6550d9167fac69552228ac39cc1a412b90fe6d0943f82c236dec8de484fb74aa"
Oct 30 22:24:42.875242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.874505    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"]
Oct 30 22:24:42.884703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:42.884670    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ca259734-openshift-default-64cb5dc965-4f6sz"]
Oct 30 22:24:44.065309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.051628    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"]
Oct 30 22:24:44.073384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.073358    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"]
Oct 30 22:24:44.073589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.073578    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec8d756-tswds"]
Oct 30 22:24:44.075461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.075435    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67206247-k5ts4"]
Oct 30 22:24:44.125088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.125049    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f284056-f73a-462b-ac37-4c245abd1486" path="/var/lib/kubelet/pods/6f284056-f73a-462b-ac37-4c245abd1486/volumes"
Oct 30 22:24:44.127051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.127027    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7acf4990-0751-41f6-ad33-ac085c2f6891" path="/var/lib/kubelet/pods/7acf4990-0751-41f6-ad33-ac085c2f6891/volumes"
Oct 30 22:24:44.129307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.128091    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" path="/var/lib/kubelet/pods/8880fff5-a9b3-4522-a4cf-8bba135c5d99/volumes"
Oct 30 22:24:44.346865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.346842    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:24:44.439460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.439423    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-podinfo\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.439672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.439479    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-socket\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.439672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.439528    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istiod-ca-cert\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.439805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.439676    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-envoy\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.439805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.439714    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-kube-api-access-f45wt\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-kube-api-access-f45wt\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.439902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.439873    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-socket" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.440250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.440206    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-data\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.440763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.440734    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-data" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.440854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.440816    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-token\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-token\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.440916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.440873    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-credential-socket\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.440986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.440962    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-certs\") pod \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\" (UID: \"e09b6a1c-f6cb-45a6-a976-417dbbc90465\") "
Oct 30 22:24:44.441205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.441058    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istiod-ca-cert" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:44.441738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.441707    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-credential-socket" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.441985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.441961    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-certs" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.442678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.442657    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.443158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.443134    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.443300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.443164    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.443300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.443179    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.443300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.443199    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.443300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.442681    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-podinfo" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:44.443300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.442698    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-envoy" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:44.443300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.443081    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-kube-api-access-f45wt" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "kube-api-access-f45wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:44.444098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.444077    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-token" pod "e09b6a1c-f6cb-45a6-a976-417dbbc90465" (UID: "e09b6a1c-f6cb-45a6-a976-417dbbc90465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:44.545174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.545136    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.545174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.545168    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.545174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.545179    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f45wt\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-kube-api-access-f45wt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:44.545428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:44.545189    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09b6a1c-f6cb-45a6-a976-417dbbc90465-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:24:45.020165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.020133    4581 generic.go:356] "Generic (PLEG): container finished" podID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerID="450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166" exitCode=0
Oct 30 22:24:45.020395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.020188    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"
Oct 30 22:24:45.020395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.020193    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" event={"ID":"e09b6a1c-f6cb-45a6-a976-417dbbc90465","Type":"ContainerDied","Data":"450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166"}
Oct 30 22:24:45.020395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.020256    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b" event={"ID":"e09b6a1c-f6cb-45a6-a976-417dbbc90465","Type":"ContainerDied","Data":"d2e693d0c1fb048cfb6d6192ee7cf6cf2e7724c8d501bdfeef9bdda16fa7fee0"}
Oct 30 22:24:45.020395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.020287    4581 scope.go:122] "RemoveContainer" containerID="450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166"
Oct 30 22:24:45.036591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.036561    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"]
Oct 30 22:24:45.037263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.037241    4581 scope.go:122] "RemoveContainer" containerID="ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc"
Oct 30 22:24:45.043644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.043619    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872ebf2a-btg2w"]
Oct 30 22:24:45.050173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.050154    4581 scope.go:122] "RemoveContainer" containerID="450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166"
Oct 30 22:24:45.050516 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:24:45.050490    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166\": container with ID starting with 450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166 not found: ID does not exist" containerID="450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166"
Oct 30 22:24:45.050568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.050529    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166"} err="failed to get container status \"450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166\": rpc error: code = NotFound desc = could not find container \"450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166\": container with ID starting with 450eeb29e868a1d9d51ecdb09e799217aa2294533f2f54120cfdeb7f0975d166 not found: ID does not exist"
Oct 30 22:24:45.050568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.050553    4581 scope.go:122] "RemoveContainer" containerID="ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc"
Oct 30 22:24:45.050774 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:24:45.050755    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc\": container with ID starting with ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc not found: ID does not exist" containerID="ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc"
Oct 30 22:24:45.050838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.050784    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc"} err="failed to get container status \"ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc\": rpc error: code = NotFound desc = could not find container \"ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc\": container with ID starting with ce6679a4dd800c59ab0c708ede5bda644653170eb89c81ea6179c5c2ff611bbc not found: ID does not exist"
Oct 30 22:24:45.077081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.077044    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"]
Oct 30 22:24:45.083017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:45.082992    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1ba9ae08-openshift-default-7758f9c76c-wbv2b"]
Oct 30 22:24:46.101890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:46.101853    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c467ff-a8f7-4274-b395-c7323ec211ec" path="/var/lib/kubelet/pods/01c467ff-a8f7-4274-b395-c7323ec211ec/volumes"
Oct 30 22:24:46.102503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:46.102485    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" path="/var/lib/kubelet/pods/e09b6a1c-f6cb-45a6-a976-417dbbc90465/volumes"
Oct 30 22:24:55.032271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:55.032234    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"]
Oct 30 22:24:55.040147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:55.040116    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d2d135-p6pfb"]
Oct 30 22:24:56.101894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:56.101854    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea62c3d-ccb5-4999-b152-04010e5487b9" path="/var/lib/kubelet/pods/2ea62c3d-ccb5-4999-b152-04010e5487b9/volumes"
Oct 30 22:24:56.383532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:24:56.383501    4581 scope.go:122] "RemoveContainer" containerID="983e316fd1f950e81a669252adbc0db3fa4a51f266a10a76cc7faf967d7f7584"
Oct 30 22:25:00.036163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:00.036121    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"]
Oct 30 22:25:00.044572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:00.044544    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bc550b-wccbz"]
Oct 30 22:25:00.099632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:00.099586    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e71675-907e-4fcd-b602-d6bdaeb1fdb6" path="/var/lib/kubelet/pods/a6e71675-907e-4fcd-b602-d6bdaeb1fdb6/volumes"
Oct 30 22:25:02.030965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:02.030928    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"]
Oct 30 22:25:02.039688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:02.039659    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1841b-bxz7k"]
Oct 30 22:25:02.103029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:02.102987    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bb5f43-ecac-4845-b8c7-69abfac4d07c" path="/var/lib/kubelet/pods/b6bb5f43-ecac-4845-b8c7-69abfac4d07c/volumes"
Oct 30 22:25:07.360245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:07.355362    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:25:07.444246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:07.443408    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnh72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:25:07.444246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:07.443524    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:25:10.821084 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:25:10.821039    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:25:15.369800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.369765    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"]
Oct 30 22:25:15.370535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370517    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy"
Oct 30 22:25:15.370535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370535    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370545    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8eaba93e-51c7-4256-9061-430629dc59d3" containerName="revoke-keys"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370550    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eaba93e-51c7-4256-9061-430629dc59d3" containerName="revoke-keys"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370565    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370570    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370577    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a39f7f88-4626-46c4-8b06-e1675ba04a54" containerName="revoke-keys"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370581    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39f7f88-4626-46c4-8b06-e1675ba04a54" containerName="revoke-keys"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370596    4581 cpu_manager.go:414] "containerMap: removing container" podUID="56bd4f0f-ed17-4042-bad4-22075314e85c" containerName="payload-pre-processing"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370601    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bd4f0f-ed17-4042-bad4-22075314e85c" containerName="payload-pre-processing"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370612    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370616    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370632    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c298b3b2-cbab-4565-a7a3-93509aca60cc" containerName="revoke-keys"
Oct 30 22:25:15.370699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370638    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c298b3b2-cbab-4565-a7a3-93509aca60cc" containerName="revoke-keys"
Oct 30 22:25:15.371589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370767    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy"
Oct 30 22:25:15.371589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370774    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eaba93e-51c7-4256-9061-430629dc59d3" containerName="revoke-keys"
Oct 30 22:25:15.371589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370781    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy"
Oct 30 22:25:15.371589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370787    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39f7f88-4626-46c4-8b06-e1675ba04a54" containerName="revoke-keys"
Oct 30 22:25:15.371589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370794    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bd4f0f-ed17-4042-bad4-22075314e85c" containerName="payload-pre-processing"
Oct 30 22:25:15.371589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370799    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8880fff5-a9b3-4522-a4cf-8bba135c5d99" containerName="istio-proxy"
Oct 30 22:25:15.371589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370806    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy"
Oct 30 22:25:15.371589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.370812    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c298b3b2-cbab-4565-a7a3-93509aca60cc" containerName="revoke-keys"
Oct 30 22:25:15.374784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.374767    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.396273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.396239    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"]
Oct 30 22:25:15.397102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.397078    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:15.397102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.397093    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:15.397341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.397106    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:25:15.397341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.397146    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:15.442905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.442857    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.443100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.442941    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tghk\" (UniqueName: \"kubernetes.io/projected/dd920218-ad0e-41a7-8fd8-b862df285f6d-kube-api-access-4tghk\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.443320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.443294    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.545365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.545301    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tghk\" (UniqueName: \"kubernetes.io/projected/dd920218-ad0e-41a7-8fd8-b862df285f6d-kube-api-access-4tghk\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.545624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.545590    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.545797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.545746    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.547547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.547527    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.552959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.552878    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.566416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.566387    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tghk\" (UniqueName: \"kubernetes.io/projected/dd920218-ad0e-41a7-8fd8-b862df285f6d-kube-api-access-4tghk\") pod \"maas-api-revoke-keys-e2e-migrate-476e593d-7ps82\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:15.694336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:15.694205    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:16.003472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:16.003442    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"]
Oct 30 22:25:16.003996 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:25:16.003973    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd920218_ad0e_41a7_8fd8_b862df285f6d.slice/crio-901d8f9a73d9cd72740168aa55666c17f3a5f1e88493b69fa2537c7bc187b81e WatchSource:0}: Error finding container 901d8f9a73d9cd72740168aa55666c17f3a5f1e88493b69fa2537c7bc187b81e: Status 404 returned error can't find the container with id 901d8f9a73d9cd72740168aa55666c17f3a5f1e88493b69fa2537c7bc187b81e
Oct 30 22:25:16.486659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:16.486622    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82" event={"ID":"dd920218-ad0e-41a7-8fd8-b862df285f6d","Type":"ContainerStarted","Data":"21fbdacfdd78c56c4ad066a327c4a8991f07e915a3adc27bdb0aa9a225c589a4"}
Oct 30 22:25:16.487084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:16.486689    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82" event={"ID":"dd920218-ad0e-41a7-8fd8-b862df285f6d","Type":"ContainerStarted","Data":"901d8f9a73d9cd72740168aa55666c17f3a5f1e88493b69fa2537c7bc187b81e"}
Oct 30 22:25:16.521498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:16.521442    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82" podStartSLOduration=1.521427163 podStartE2EDuration="1.521427163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:16.519494876 +0000 UTC m=+1100.558433761" watchObservedRunningTime="2026-10-30 22:25:16.521427163 +0000 UTC m=+1100.560366098"
Oct 30 22:25:17.515204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:17.515160    4581 generic.go:356] "Generic (PLEG): container finished" podID="dd920218-ad0e-41a7-8fd8-b862df285f6d" containerID="21fbdacfdd78c56c4ad066a327c4a8991f07e915a3adc27bdb0aa9a225c589a4" exitCode=0
Oct 30 22:25:17.515781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:17.515262    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82" event={"ID":"dd920218-ad0e-41a7-8fd8-b862df285f6d","Type":"ContainerDied","Data":"21fbdacfdd78c56c4ad066a327c4a8991f07e915a3adc27bdb0aa9a225c589a4"}
Oct 30 22:25:18.728564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.728538    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:18.809037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.808956    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd920218-ad0e-41a7-8fd8-b862df285f6d-kube-api-access-4tghk\" (UniqueName: \"kubernetes.io/projected/dd920218-ad0e-41a7-8fd8-b862df285f6d-kube-api-access-4tghk\") pod \"dd920218-ad0e-41a7-8fd8-b862df285f6d\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") "
Oct 30 22:25:18.809283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.809261    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-xks-ca\") pod \"dd920218-ad0e-41a7-8fd8-b862df285f6d\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") "
Oct 30 22:25:18.809397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.809382    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-openshift-ca\") pod \"dd920218-ad0e-41a7-8fd8-b862df285f6d\" (UID: \"dd920218-ad0e-41a7-8fd8-b862df285f6d\") "
Oct 30 22:25:18.809855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.809819    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-openshift-ca" pod "dd920218-ad0e-41a7-8fd8-b862df285f6d" (UID: "dd920218-ad0e-41a7-8fd8-b862df285f6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:18.810150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.810125    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:18.811445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.811419    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd920218-ad0e-41a7-8fd8-b862df285f6d-kube-api-access-4tghk" pod "dd920218-ad0e-41a7-8fd8-b862df285f6d" (UID: "dd920218-ad0e-41a7-8fd8-b862df285f6d"). InnerVolumeSpecName "kube-api-access-4tghk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:18.811562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.811540    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-xks-ca" pod "dd920218-ad0e-41a7-8fd8-b862df285f6d" (UID: "dd920218-ad0e-41a7-8fd8-b862df285f6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:18.911906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.911869    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd920218-ad0e-41a7-8fd8-b862df285f6d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:18.911906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:18.911902    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tghk\" (UniqueName: \"kubernetes.io/projected/dd920218-ad0e-41a7-8fd8-b862df285f6d-kube-api-access-4tghk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:19.537790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:19.537758    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"
Oct 30 22:25:19.537946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:19.537756    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82" event={"ID":"dd920218-ad0e-41a7-8fd8-b862df285f6d","Type":"ContainerDied","Data":"901d8f9a73d9cd72740168aa55666c17f3a5f1e88493b69fa2537c7bc187b81e"}
Oct 30 22:25:19.537946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:19.537895    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="901d8f9a73d9cd72740168aa55666c17f3a5f1e88493b69fa2537c7bc187b81e"
Oct 30 22:25:23.133465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.133421    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"]
Oct 30 22:25:23.134132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.134118    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy"
Oct 30 22:25:23.134179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.134133    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09b6a1c-f6cb-45a6-a976-417dbbc90465" containerName="istio-proxy"
Oct 30 22:25:23.134179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.134148    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dd920218-ad0e-41a7-8fd8-b862df285f6d" containerName="revoke-keys"
Oct 30 22:25:23.134179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.134155    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd920218-ad0e-41a7-8fd8-b862df285f6d" containerName="revoke-keys"
Oct 30 22:25:23.134320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.134294    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd920218-ad0e-41a7-8fd8-b862df285f6d" containerName="revoke-keys"
Oct 30 22:25:23.138317 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.138301    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.167473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.161980    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:23.167473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.162182    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:25:23.167473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.162287    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:23.167473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.162317    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:23.195138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.195105    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"]
Oct 30 22:25:23.289695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.289640    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9djb\" (UniqueName: \"kubernetes.io/projected/360edc1a-3b2f-475c-91a0-b90dc9b895ab-kube-api-access-z9djb\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.289992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.289964    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.290077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.290027    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.392113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.392020    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9djb\" (UniqueName: \"kubernetes.io/projected/360edc1a-3b2f-475c-91a0-b90dc9b895ab-kube-api-access-z9djb\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.392113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.392083    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.392354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.392120    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.392902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.392837    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.394728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.394702    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.409605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.409573    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9djb\" (UniqueName: \"kubernetes.io/projected/360edc1a-3b2f-475c-91a0-b90dc9b895ab-kube-api-access-z9djb\") pod \"maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:23.495101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:23.495066    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:24.069587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:24.069554    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"]
Oct 30 22:25:24.070929 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:25:24.070888    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360edc1a_3b2f_475c_91a0_b90dc9b895ab.slice/crio-a7fd0a2de3d545b5b158c25124605fb69984727afa8501911c6f76b054c5e95c WatchSource:0}: Error finding container a7fd0a2de3d545b5b158c25124605fb69984727afa8501911c6f76b054c5e95c: Status 404 returned error can't find the container with id a7fd0a2de3d545b5b158c25124605fb69984727afa8501911c6f76b054c5e95c
Oct 30 22:25:24.630053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:24.630017    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4" event={"ID":"360edc1a-3b2f-475c-91a0-b90dc9b895ab","Type":"ContainerStarted","Data":"98369fc84b57406ed73afe90d10bb48fdeab304dc192a50c4e782e239a7a238e"}
Oct 30 22:25:24.630554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:24.630074    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4" event={"ID":"360edc1a-3b2f-475c-91a0-b90dc9b895ab","Type":"ContainerStarted","Data":"a7fd0a2de3d545b5b158c25124605fb69984727afa8501911c6f76b054c5e95c"}
Oct 30 22:25:24.754683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:24.747278    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4" podStartSLOduration=1.747253561 podStartE2EDuration="1.747253561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:24.695692666 +0000 UTC m=+1108.734631576" watchObservedRunningTime="2026-10-30 22:25:24.747253561 +0000 UTC m=+1108.786192448"
Oct 30 22:25:24.794210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:24.794135    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"]
Oct 30 22:25:24.938634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:24.938545    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"]
Oct 30 22:25:24.940113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:24.938681    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.041019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.040971    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.041249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.041055    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.041249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.041188    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf9lq\" (UniqueName: \"kubernetes.io/projected/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-kube-api-access-wf9lq\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.143092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.143060    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.143318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.143117    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.143318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.143149    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9lq\" (UniqueName: \"kubernetes.io/projected/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-kube-api-access-wf9lq\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.143836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.143809    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.145904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.145880    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.160788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.160758    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9lq\" (UniqueName: \"kubernetes.io/projected/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-kube-api-access-wf9lq\") pod \"maas-api-revoke-sub-4d52998c-xj82m\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.264728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.264641    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:25.595546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.595513    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"]
Oct 30 22:25:25.596613 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:25:25.596583    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07b41a9_3afc_443c_b38e_bb8fcd634f3c.slice/crio-e56eddd7392740440343430364ea9dff60e2197c6dbb2ddd2de567bb7be5ec2b WatchSource:0}: Error finding container e56eddd7392740440343430364ea9dff60e2197c6dbb2ddd2de567bb7be5ec2b: Status 404 returned error can't find the container with id e56eddd7392740440343430364ea9dff60e2197c6dbb2ddd2de567bb7be5ec2b
Oct 30 22:25:25.643111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.643072    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m" event={"ID":"f07b41a9-3afc-443c-b38e-bb8fcd634f3c","Type":"ContainerStarted","Data":"e56eddd7392740440343430364ea9dff60e2197c6dbb2ddd2de567bb7be5ec2b"}
Oct 30 22:25:25.645083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.645056    4581 generic.go:356] "Generic (PLEG): container finished" podID="360edc1a-3b2f-475c-91a0-b90dc9b895ab" containerID="98369fc84b57406ed73afe90d10bb48fdeab304dc192a50c4e782e239a7a238e" exitCode=0
Oct 30 22:25:25.645183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:25.645128    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4" event={"ID":"360edc1a-3b2f-475c-91a0-b90dc9b895ab","Type":"ContainerDied","Data":"98369fc84b57406ed73afe90d10bb48fdeab304dc192a50c4e782e239a7a238e"}
Oct 30 22:25:26.743810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:26.743763    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m" event={"ID":"f07b41a9-3afc-443c-b38e-bb8fcd634f3c","Type":"ContainerStarted","Data":"397917c9499f49344afe0572cf91608efac089de861eb4dda072485a37d53526"}
Oct 30 22:25:26.785985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:26.785928    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m" podStartSLOduration=2.785911676 podStartE2EDuration="2.785911676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:26.778648546 +0000 UTC m=+1110.817587431" watchObservedRunningTime="2026-10-30 22:25:26.785911676 +0000 UTC m=+1110.824850618"
Oct 30 22:25:27.021773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.021697    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:27.101257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.101208    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-openshift-ca\") pod \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") "
Oct 30 22:25:27.101491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.101471    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360edc1a-3b2f-475c-91a0-b90dc9b895ab-kube-api-access-z9djb\" (UniqueName: \"kubernetes.io/projected/360edc1a-3b2f-475c-91a0-b90dc9b895ab-kube-api-access-z9djb\") pod \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") "
Oct 30 22:25:27.101577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.101558    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-openshift-ca" pod "360edc1a-3b2f-475c-91a0-b90dc9b895ab" (UID: "360edc1a-3b2f-475c-91a0-b90dc9b895ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:27.101635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.101585    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-xks-ca\") pod \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\" (UID: \"360edc1a-3b2f-475c-91a0-b90dc9b895ab\") "
Oct 30 22:25:27.102621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.102587    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:27.103885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.103865    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360edc1a-3b2f-475c-91a0-b90dc9b895ab-kube-api-access-z9djb" pod "360edc1a-3b2f-475c-91a0-b90dc9b895ab" (UID: "360edc1a-3b2f-475c-91a0-b90dc9b895ab"). InnerVolumeSpecName "kube-api-access-z9djb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:27.104355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.104332    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-xks-ca" pod "360edc1a-3b2f-475c-91a0-b90dc9b895ab" (UID: "360edc1a-3b2f-475c-91a0-b90dc9b895ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:27.204306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.204264    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9djb\" (UniqueName: \"kubernetes.io/projected/360edc1a-3b2f-475c-91a0-b90dc9b895ab-kube-api-access-z9djb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:27.204306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.204303    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360edc1a-3b2f-475c-91a0-b90dc9b895ab-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:27.754097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.754060    4581 generic.go:356] "Generic (PLEG): container finished" podID="f07b41a9-3afc-443c-b38e-bb8fcd634f3c" containerID="397917c9499f49344afe0572cf91608efac089de861eb4dda072485a37d53526" exitCode=0
Oct 30 22:25:27.754575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.754147    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m" event={"ID":"f07b41a9-3afc-443c-b38e-bb8fcd634f3c","Type":"ContainerDied","Data":"397917c9499f49344afe0572cf91608efac089de861eb4dda072485a37d53526"}
Oct 30 22:25:27.756152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.756127    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"
Oct 30 22:25:27.756152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.756141    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4" event={"ID":"360edc1a-3b2f-475c-91a0-b90dc9b895ab","Type":"ContainerDied","Data":"a7fd0a2de3d545b5b158c25124605fb69984727afa8501911c6f76b054c5e95c"}
Oct 30 22:25:27.756392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:27.756186    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7fd0a2de3d545b5b158c25124605fb69984727afa8501911c6f76b054c5e95c"
Oct 30 22:25:28.969601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:28.969559    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:29.035807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.035724    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-openshift-ca\") pod \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") "
Oct 30 22:25:29.035986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.035878    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-xks-ca\") pod \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") "
Oct 30 22:25:29.035986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.035918    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-kube-api-access-wf9lq\" (UniqueName: \"kubernetes.io/projected/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-kube-api-access-wf9lq\") pod \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\" (UID: \"f07b41a9-3afc-443c-b38e-bb8fcd634f3c\") "
Oct 30 22:25:29.036155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.036102    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-openshift-ca" pod "f07b41a9-3afc-443c-b38e-bb8fcd634f3c" (UID: "f07b41a9-3afc-443c-b38e-bb8fcd634f3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:29.036696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.036665    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:29.038238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.038190    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-xks-ca" pod "f07b41a9-3afc-443c-b38e-bb8fcd634f3c" (UID: "f07b41a9-3afc-443c-b38e-bb8fcd634f3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:29.038873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.038850    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-kube-api-access-wf9lq" pod "f07b41a9-3afc-443c-b38e-bb8fcd634f3c" (UID: "f07b41a9-3afc-443c-b38e-bb8fcd634f3c"). InnerVolumeSpecName "kube-api-access-wf9lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:29.138817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.138775    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:29.138817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.138805    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf9lq\" (UniqueName: \"kubernetes.io/projected/f07b41a9-3afc-443c-b38e-bb8fcd634f3c-kube-api-access-wf9lq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:25:29.783138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.783098    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"
Oct 30 22:25:29.783138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.783118    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m" event={"ID":"f07b41a9-3afc-443c-b38e-bb8fcd634f3c","Type":"ContainerDied","Data":"e56eddd7392740440343430364ea9dff60e2197c6dbb2ddd2de567bb7be5ec2b"}
Oct 30 22:25:29.783394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:29.783172    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e56eddd7392740440343430364ea9dff60e2197c6dbb2ddd2de567bb7be5ec2b"
Oct 30 22:25:33.023330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:33.023298    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"]
Oct 30 22:25:33.029301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:33.029269    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-476e593d-7ps82"]
Oct 30 22:25:34.178518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:34.178177    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd920218-ad0e-41a7-8fd8-b862df285f6d" path="/var/lib/kubelet/pods/dd920218-ad0e-41a7-8fd8-b862df285f6d/volumes"
Oct 30 22:25:37.421674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:37.421584    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"]
Oct 30 22:25:37.492246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:37.442264    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-913eddcb-tp7k4"]
Oct 30 22:25:38.100496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:38.100462    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360edc1a-3b2f-475c-91a0-b90dc9b895ab" path="/var/lib/kubelet/pods/360edc1a-3b2f-475c-91a0-b90dc9b895ab/volumes"
Oct 30 22:25:40.824093 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:25:40.824052    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:25:40.861770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.861732    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"]
Oct 30 22:25:40.862562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.862542    4581 cpu_manager.go:414] "containerMap: removing container" podUID="360edc1a-3b2f-475c-91a0-b90dc9b895ab" containerName="revoke-keys"
Oct 30 22:25:40.862562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.862562    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360edc1a-3b2f-475c-91a0-b90dc9b895ab" containerName="revoke-keys"
Oct 30 22:25:40.862716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.862593    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f07b41a9-3afc-443c-b38e-bb8fcd634f3c" containerName="revoke-keys"
Oct 30 22:25:40.862716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.862598    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07b41a9-3afc-443c-b38e-bb8fcd634f3c" containerName="revoke-keys"
Oct 30 22:25:40.862716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.862700    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="360edc1a-3b2f-475c-91a0-b90dc9b895ab" containerName="revoke-keys"
Oct 30 22:25:40.862716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.862714    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07b41a9-3afc-443c-b38e-bb8fcd634f3c" containerName="revoke-keys"
Oct 30 22:25:40.868039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.868021    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:40.888699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.888660    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"]
Oct 30 22:25:40.888960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.888930    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-80ecb3\""
Oct 30 22:25:40.999901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:40.999862    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592319f-123c-4546-8a47-668a8fb03851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg\" (UID: \"6592319f-123c-4546-8a47-668a8fb03851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:41.000318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:41.000282    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crqqv\" (UniqueName: \"kubernetes.io/projected/6592319f-123c-4546-8a47-668a8fb03851-kube-api-access-crqqv\") pod \"payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg\" (UID: \"6592319f-123c-4546-8a47-668a8fb03851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:41.119321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:41.104275    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592319f-123c-4546-8a47-668a8fb03851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg\" (UID: \"6592319f-123c-4546-8a47-668a8fb03851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:41.119321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:41.105393    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592319f-123c-4546-8a47-668a8fb03851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg\" (UID: \"6592319f-123c-4546-8a47-668a8fb03851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:41.119321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:41.105769    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crqqv\" (UniqueName: \"kubernetes.io/projected/6592319f-123c-4546-8a47-668a8fb03851-kube-api-access-crqqv\") pod \"payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg\" (UID: \"6592319f-123c-4546-8a47-668a8fb03851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:41.133331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:41.126115    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crqqv\" (UniqueName: \"kubernetes.io/projected/6592319f-123c-4546-8a47-668a8fb03851-kube-api-access-crqqv\") pod \"payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg\" (UID: \"6592319f-123c-4546-8a47-668a8fb03851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:41.195245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:41.192853    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:41.602429 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:25:41.599906    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6592319f_123c_4546_8a47_668a8fb03851.slice/crio-9ba38c6fe72570a62d7c62b1a23f3afa73f37cc29ede898376efdd1027898072 WatchSource:0}: Error finding container 9ba38c6fe72570a62d7c62b1a23f3afa73f37cc29ede898376efdd1027898072: Status 404 returned error can't find the container with id 9ba38c6fe72570a62d7c62b1a23f3afa73f37cc29ede898376efdd1027898072
Oct 30 22:25:41.607958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:41.607925    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"]
Oct 30 22:25:42.019867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:42.019826    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg" event={"ID":"6592319f-123c-4546-8a47-668a8fb03851","Type":"ContainerStarted","Data":"6baeb8ca5cd1ad6b028ab5146ebc5d04e931a2eb73aad08b8b92f8e61ca62d67"}
Oct 30 22:25:42.020446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:42.019886    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg" event={"ID":"6592319f-123c-4546-8a47-668a8fb03851","Type":"ContainerStarted","Data":"9ba38c6fe72570a62d7c62b1a23f3afa73f37cc29ede898376efdd1027898072"}
Oct 30 22:25:42.020446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:42.020043    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:42.058132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:42.058060    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg" podStartSLOduration=2.058040176 podStartE2EDuration="2.058040176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:42.054730748 +0000 UTC m=+1126.093669634" watchObservedRunningTime="2026-10-30 22:25:42.058040176 +0000 UTC m=+1126.096979065"
Oct 30 22:25:53.038013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:53.037983    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 22:25:56.463658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.463621    4581 scope.go:122] "RemoveContainer" containerID="b58e5aca517c1b5db48b1b39b8d68193e9713d26afbcc17327d0d6f0c6a35b59"
Oct 30 22:25:56.482460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.482435    4581 scope.go:122] "RemoveContainer" containerID="1472ba5389f0173f1b49986ba1cf72400cae06edef154e336c984f4db6363fff"
Oct 30 22:25:56.495829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.495803    4581 scope.go:122] "RemoveContainer" containerID="3e0f1db2b281bf9f05f0b07270ca381e4356c27d581f1cc28469afef18239bc9"
Oct 30 22:25:56.510067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.510041    4581 scope.go:122] "RemoveContainer" containerID="83c1d20857a2b4f1aa4beddda0a5ae840ca7236573f3ff6d24b2f809ddc75e88"
Oct 30 22:25:56.526465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.526442    4581 scope.go:122] "RemoveContainer" containerID="c56e27c8a6affdbba67e1cba124d3446921d088ce32b505905657f654d267a15"
Oct 30 22:25:56.540671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.540644    4581 scope.go:122] "RemoveContainer" containerID="427828c0f7434cc762367c0077183ba5449823a68b23d6e2265eb6c057656064"
Oct 30 22:25:56.569267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.569088    4581 scope.go:122] "RemoveContainer" containerID="925368464c7e84ea8b61a2884378efc2997f222072475cd4bff4ed9c3af09f26"
Oct 30 22:25:56.590804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.590470    4581 scope.go:122] "RemoveContainer" containerID="41a57df7a8cbc97a1916fe90ad437b04aaf79f93032e1488bdf366a6653ee32e"
Oct 30 22:25:56.702708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.702656    4581 scope.go:122] "RemoveContainer" containerID="c6f8d87b4c2cb04684d44b9fb4d1a87ec2a2bf69d17a8aefcb7e75252442a096"
Oct 30 22:25:56.734376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.732840    4581 scope.go:122] "RemoveContainer" containerID="b16170679b146db1c43477b03c87ee33d9b6276a337e0bec876f8e689c889f77"
Oct 30 22:25:56.776657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.776629    4581 scope.go:122] "RemoveContainer" containerID="df31263be56d90577547bfbc16dffbbd48fc8e0cce7920f496c8d7332ec71c94"
Oct 30 22:25:56.877556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.877268    4581 scope.go:122] "RemoveContainer" containerID="f475c44e5c69a0f00d61aa9fc4259627b5b45375128388f00cd452d429ca56e4"
Oct 30 22:25:56.966290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:56.966252    4581 scope.go:122] "RemoveContainer" containerID="9671168a4299ffca9a22726fa6f787ad542b1668bd522e848a14ae94d1e82975"
Oct 30 22:25:57.002867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:57.000231    4581 scope.go:122] "RemoveContainer" containerID="8b25fd899688b0bd0237b2958549e0925fca012e30cce7a23fbe55f3772a4318"
Oct 30 22:25:57.027725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:57.027568    4581 scope.go:122] "RemoveContainer" containerID="731a6b3ec2829fc79c6e85723372b73fee3dd24897b709532614ff1b9ff9f695"
Oct 30 22:25:57.058526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:57.055783    4581 scope.go:122] "RemoveContainer" containerID="5529460fae754847dd23316eba7a1fe13c740f41160c5a20510949b383a3a4ee"
Oct 30 22:25:58.459381 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:58.459325    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:25:58.653244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:58.642664    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:25:58.754721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:58.754620    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:25:59.040304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:59.036789    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:25:59.076204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:25:59.075990    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 30 22:26:10.819602 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:26:10.819550    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:26:20.397773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.397644    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"]
Oct 30 22:26:20.404169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.404136    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.426190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.426151    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"]
Oct 30 22:26:20.427262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.427196    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3cd77f-openshift-default-dockercfg-vlt6j\""
Oct 30 22:26:20.469618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.469567    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.469829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.469632    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.469829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.469659    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.469829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.469753    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.469829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.469821    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.470055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.469889    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043e7603-d47f-43c3-aee4-a62b9c1922d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.470055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.469971    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.470055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.470039    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.470205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.470100    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6pg\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-kube-api-access-4t6pg\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.574312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.574270    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.574633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.574616    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.574804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.574776    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6pg\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-kube-api-access-4t6pg\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.574962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.574943    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.575026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.574980    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.575026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.575005    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.575143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.575102    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.575209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.575175    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.575311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.575270    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043e7603-d47f-43c3-aee4-a62b9c1922d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.576145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.576118    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043e7603-d47f-43c3-aee4-a62b9c1922d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.598251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.591538    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.598251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.591787    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.598251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.592343    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.598251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.593060    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.606518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.606487    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.611251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.608914    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.615270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.614117    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.627833 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.616913    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6pg\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-kube-api-access-4t6pg\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:20.737421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:20.736588    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:21.196396 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:26:21.196361    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043e7603_d47f_43c3_aee4_a62b9c1922d2.slice/crio-416686301e1c63e7ec017639939be9afb7c86c43d53187f5de159e6d280d9ad5 WatchSource:0}: Error finding container 416686301e1c63e7ec017639939be9afb7c86c43d53187f5de159e6d280d9ad5: Status 404 returned error can't find the container with id 416686301e1c63e7ec017639939be9afb7c86c43d53187f5de159e6d280d9ad5
Oct 30 22:26:21.197351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:21.197324    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"]
Oct 30 22:26:21.199109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:21.199065    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:21.199208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:21.199157    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:21.199271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:21.199211    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:26:21.420323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:21.420276    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" event={"ID":"043e7603-d47f-43c3-aee4-a62b9c1922d2","Type":"ContainerStarted","Data":"70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5"}
Oct 30 22:26:21.420795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:21.420351    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" event={"ID":"043e7603-d47f-43c3-aee4-a62b9c1922d2","Type":"ContainerStarted","Data":"416686301e1c63e7ec017639939be9afb7c86c43d53187f5de159e6d280d9ad5"}
Oct 30 22:26:21.467571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:21.467447    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podStartSLOduration=1.467426755 podStartE2EDuration="1.467426755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:21.457915751 +0000 UTC m=+1165.496854639" watchObservedRunningTime="2026-10-30 22:26:21.467426755 +0000 UTC m=+1165.506365642"
Oct 30 22:26:21.737072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:21.736989    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:22.743019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:22.742987    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:23.439937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:23.439905    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:23.441083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:23.441064    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:26:24.031382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:24.031341    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"]
Oct 30 22:26:24.040052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:24.040023    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6616045-npdqg"]
Oct 30 22:26:24.101875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:24.101837    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4087e96b-4767-4fb0-a808-fee382dedb69" path="/var/lib/kubelet/pods/4087e96b-4767-4fb0-a808-fee382dedb69/volumes"
Oct 30 22:26:36.916279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:36.916237    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"]
Oct 30 22:26:36.921426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:36.921402    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:36.936420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:36.936395    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6af86e-dockercfg-6kzwt\""
Oct 30 22:26:36.943414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:36.943385    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"]
Oct 30 22:26:36.946181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:36.946156    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6af86e\""
Oct 30 22:26:37.099681 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:37.079112    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zcch\" (UniqueName: \"kubernetes.io/projected/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-kube-api-access-5zcch\") pod \"payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw\" (UID: \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:37.123903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:37.123865    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw\" (UID: \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:37.256816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:37.255640    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw\" (UID: \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:37.256816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:37.256103    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zcch\" (UniqueName: \"kubernetes.io/projected/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-kube-api-access-5zcch\") pod \"payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw\" (UID: \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:37.256816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:37.256568    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw\" (UID: \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:37.273554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:37.273520    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zcch\" (UniqueName: \"kubernetes.io/projected/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-kube-api-access-5zcch\") pod \"payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw\" (UID: \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:37.539542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:37.539461    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:37.849511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:37.849478    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"]
Oct 30 22:26:37.849800 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:26:37.849776    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8c98e1_9d5b_48f1_af1f_5b1ed4afab4a.slice/crio-57fff09bc2dfafc73259676787673b4489d13bdd619153f0055082eb18f248a9 WatchSource:0}: Error finding container 57fff09bc2dfafc73259676787673b4489d13bdd619153f0055082eb18f248a9: Status 404 returned error can't find the container with id 57fff09bc2dfafc73259676787673b4489d13bdd619153f0055082eb18f248a9
Oct 30 22:26:38.031084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:38.031051    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"]
Oct 30 22:26:38.040154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:38.040127    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c5366d-cbcpg"]
Oct 30 22:26:38.099284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:38.099174    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106ff678-a388-4db1-8947-d5c834a140c7" path="/var/lib/kubelet/pods/106ff678-a388-4db1-8947-d5c834a140c7/volumes"
Oct 30 22:26:38.751190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:38.751157    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw" event={"ID":"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a","Type":"ContainerStarted","Data":"24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b"}
Oct 30 22:26:38.751425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:38.751259    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw" event={"ID":"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a","Type":"ContainerStarted","Data":"57fff09bc2dfafc73259676787673b4489d13bdd619153f0055082eb18f248a9"}
Oct 30 22:26:38.751425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:38.751294    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:38.788980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:38.788916    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw" podStartSLOduration=2.788899314 podStartE2EDuration="2.788899314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:38.785544929 +0000 UTC m=+1182.824483815" watchObservedRunningTime="2026-10-30 22:26:38.788899314 +0000 UTC m=+1182.827838200"
Oct 30 22:26:40.819578 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:26:40.819527    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:26:48.031880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:48.031842    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"]
Oct 30 22:26:48.040876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:48.040845    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d20ee0-z65zm"]
Oct 30 22:26:48.099459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:48.099424    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be5f1ed-bc46-45cb-a778-bbb6e57db950" path="/var/lib/kubelet/pods/9be5f1ed-bc46-45cb-a778-bbb6e57db950/volumes"
Oct 30 22:26:49.031124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:49.031085    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"]
Oct 30 22:26:49.039846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:49.039808    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86bee085-8crdg"]
Oct 30 22:26:49.764597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:49.764569    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 22:26:50.101960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:50.101878    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf94f1e0-8df4-409d-adcb-7d0423161a7f" path="/var/lib/kubelet/pods/bf94f1e0-8df4-409d-adcb-7d0423161a7f/volumes"
Oct 30 22:26:52.031272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:52.031233    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"]
Oct 30 22:26:52.040604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:52.040571    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-437503af-9v7cl"]
Oct 30 22:26:52.102337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:52.102299    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27af7364-b7f5-407f-8b92-d6a55ba6a092" path="/var/lib/kubelet/pods/27af7364-b7f5-407f-8b92-d6a55ba6a092/volumes"
Oct 30 22:26:56.837873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:56.837842    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:26:56.841960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:56.841937    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:26:57.561780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:57.561749    4581 scope.go:122] "RemoveContainer" containerID="a18ca81caea4b67cc3c284c721fae7a997ad029d9f7458154dd166a5ebea685a"
Oct 30 22:26:57.575993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:57.575965    4581 scope.go:122] "RemoveContainer" containerID="7bc2e614c5d1d4bc878879545eacb614b5a9fa26b230ab7826b171a12805adff"
Oct 30 22:26:59.031442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:59.031399    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"]
Oct 30 22:26:59.041768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:26:59.041727    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822617f2-dpwmq"]
Oct 30 22:27:00.102082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:00.102043    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c1f420-d3ae-4f67-ac70-e8bbdf282a42" path="/var/lib/kubelet/pods/00c1f420-d3ae-4f67-ac70-e8bbdf282a42/volumes"
Oct 30 22:27:06.364334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.364230    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"]
Oct 30 22:27:06.371373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.371341    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.387524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.387478    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d399fb07-openshift-default-dockercfg-gkn9c\""
Oct 30 22:27:06.409100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.409064    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"]
Oct 30 22:27:06.492617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.492571    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.492803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.492720    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d11d3-326a-4c67-817a-7b9a7cdf9596-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.492893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.492868    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.492943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.492927    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.493028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.493010    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk72x\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-kube-api-access-gk72x\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.493112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.493096    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.493325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.493300    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.493543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.493522    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.493630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.493563    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.611250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.607770    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.611250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.608394    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.612403    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d11d3-326a-4c67-817a-7b9a7cdf9596-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.612546    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.612596    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.612632    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk72x\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-kube-api-access-gk72x\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.612727    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.612831    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.612954    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.612995    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.614886    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d11d3-326a-4c67-817a-7b9a7cdf9596-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.615439    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.617313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.615697    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.618442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.618417    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.632450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.632421    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.634980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.634950    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk72x\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-kube-api-access-gk72x\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.635191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.634963    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.641240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.641161    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:06.708892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:06.697573    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:07.103571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:07.103534    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"]
Oct 30 22:27:07.105273 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:27:07.105241    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043d11d3_326a_4c67_817a_7b9a7cdf9596.slice/crio-f38f31828e9b3c77ec0f63d75d5b92265770fcc495eb1479ed14666534cd5d9e WatchSource:0}: Error finding container f38f31828e9b3c77ec0f63d75d5b92265770fcc495eb1479ed14666534cd5d9e: Status 404 returned error can't find the container with id f38f31828e9b3c77ec0f63d75d5b92265770fcc495eb1479ed14666534cd5d9e
Oct 30 22:27:07.107998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:07.107964    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:07.108060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:07.108037    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:07.108097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:07.108067    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:08.117856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:08.117821    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" event={"ID":"043d11d3-326a-4c67-817a-7b9a7cdf9596","Type":"ContainerStarted","Data":"e7480b2d393ec706c625bb4f3e72c7b51aef309dd99879cce3531c91ecd09408"}
Oct 30 22:27:08.118320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:08.117873    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" event={"ID":"043d11d3-326a-4c67-817a-7b9a7cdf9596","Type":"ContainerStarted","Data":"f38f31828e9b3c77ec0f63d75d5b92265770fcc495eb1479ed14666534cd5d9e"}
Oct 30 22:27:08.171954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:08.171895    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podStartSLOduration=2.171879922 podStartE2EDuration="2.171879922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:08.165576654 +0000 UTC m=+1212.204515540" watchObservedRunningTime="2026-10-30 22:27:08.171879922 +0000 UTC m=+1212.210818808"
Oct 30 22:27:08.698120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:08.698080    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:08.705033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:08.705003    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:09.133428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.133388    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:09.134389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.134370    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:27:09.641542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.641491    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"]
Oct 30 22:27:09.648307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.648268    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.662265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.661465    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-6f2eba-openshift-default-dockercfg-kkm8p\""
Oct 30 22:27:09.682208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.682169    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"]
Oct 30 22:27:09.736096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736056    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istiod-ca-cert\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.736306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736161    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-certs\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.736362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736341    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-envoy\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.736507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736486    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-podinfo\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.736551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736536    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-data\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.736590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736578    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-credential-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.736634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736611    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-token\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.736691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736676    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.736727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.736704    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8bn\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-kube-api-access-lv8bn\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.840498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840456    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istiod-ca-cert\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.840778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840530    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-certs\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.840778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840669    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-envoy\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.840926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840781    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-podinfo\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.840926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840819    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-data\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.840926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840854    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-credential-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.840926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840885    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-token\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.841144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840945    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.841144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.840973    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8bn\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-kube-api-access-lv8bn\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.842163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.842130    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istiod-ca-cert\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.843674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.843635    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-data\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.849181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.847617    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-envoy\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.849181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.848186    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-credential-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.849181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.848454    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-socket\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.875324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.870906    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-token\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.875324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.874520    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-certs\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.904619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.883363    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8bn\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-kube-api-access-lv8bn\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:09.905416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:09.905385    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-podinfo\") pod \"e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") " pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:10.005260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:10.003822    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:10.377794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:10.377764    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"]
Oct 30 22:27:10.378176 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:27:10.377907    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead9f4b6_1d08_4006_a8a9_e99ebcca9ac1.slice/crio-68b4ddff9d7e3b332c394f68a633d26198171cefb52c6ec579d060abf119d51e WatchSource:0}: Error finding container 68b4ddff9d7e3b332c394f68a633d26198171cefb52c6ec579d060abf119d51e: Status 404 returned error can't find the container with id 68b4ddff9d7e3b332c394f68a633d26198171cefb52c6ec579d060abf119d51e
Oct 30 22:27:10.380715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:10.380680    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:10.380812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:10.380751    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:10.380812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:10.380781    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:11.019042 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:27:11.019001    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:27:11.198947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:11.198908    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" event={"ID":"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1","Type":"ContainerStarted","Data":"cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5"}
Oct 30 22:27:11.199163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:11.198971    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" event={"ID":"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1","Type":"ContainerStarted","Data":"68b4ddff9d7e3b332c394f68a633d26198171cefb52c6ec579d060abf119d51e"}
Oct 30 22:27:11.252640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:11.252576    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podStartSLOduration=2.252559752 podStartE2EDuration="2.252559752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:11.245030029 +0000 UTC m=+1215.283968914" watchObservedRunningTime="2026-10-30 22:27:11.252559752 +0000 UTC m=+1215.291498703"
Oct 30 22:27:12.005776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.005742    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:12.011690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.011664    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:12.213581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.213545    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:12.214799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.214773    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:27:12.418451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.418416    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"]
Oct 30 22:27:12.423506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.423477    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:12.447322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.447087    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"]
Oct 30 22:27:12.497015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.496545    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d399fb07\""
Oct 30 22:27:12.497015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.496679    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d399fb07-dockercfg-mrr66\""
Oct 30 22:27:12.559247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.558526    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9ce949-0e67-4b71-8977-94cd796853cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4\" (UID: \"ec9ce949-0e67-4b71-8977-94cd796853cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:12.559247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.558745    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbvl\" (UniqueName: \"kubernetes.io/projected/ec9ce949-0e67-4b71-8977-94cd796853cb-kube-api-access-phbvl\") pod \"payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4\" (UID: \"ec9ce949-0e67-4b71-8977-94cd796853cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:12.717297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.715871    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbvl\" (UniqueName: \"kubernetes.io/projected/ec9ce949-0e67-4b71-8977-94cd796853cb-kube-api-access-phbvl\") pod \"payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4\" (UID: \"ec9ce949-0e67-4b71-8977-94cd796853cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:12.717297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.716164    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9ce949-0e67-4b71-8977-94cd796853cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4\" (UID: \"ec9ce949-0e67-4b71-8977-94cd796853cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:12.717536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.717297    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9ce949-0e67-4b71-8977-94cd796853cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4\" (UID: \"ec9ce949-0e67-4b71-8977-94cd796853cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:12.768247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:12.768109    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbvl\" (UniqueName: \"kubernetes.io/projected/ec9ce949-0e67-4b71-8977-94cd796853cb-kube-api-access-phbvl\") pod \"payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4\" (UID: \"ec9ce949-0e67-4b71-8977-94cd796853cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:13.029829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:13.029745    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"]
Oct 30 22:27:13.035672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:13.035645    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa65e8a4-dzlvf"]
Oct 30 22:27:13.062573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:13.062534    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:13.378000 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:27:13.377962    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9ce949_0e67_4b71_8977_94cd796853cb.slice/crio-e39ea75b80e21c1f947ec4ce89da64ce146e28d19a7533ce962f6eead57beba8 WatchSource:0}: Error finding container e39ea75b80e21c1f947ec4ce89da64ce146e28d19a7533ce962f6eead57beba8: Status 404 returned error can't find the container with id e39ea75b80e21c1f947ec4ce89da64ce146e28d19a7533ce962f6eead57beba8
Oct 30 22:27:13.380682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:13.380656    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"]
Oct 30 22:27:14.038892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:14.038854    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"]
Oct 30 22:27:14.054290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:14.054241    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bb8d901-q5mdr"]
Oct 30 22:27:14.106937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:14.105968    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd0f3ad-37c2-42b0-a475-5ef1ada45958" path="/var/lib/kubelet/pods/dcd0f3ad-37c2-42b0-a475-5ef1ada45958/volumes"
Oct 30 22:27:14.107139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:14.106960    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed77a059-e350-4394-9e32-0901e740b159" path="/var/lib/kubelet/pods/ed77a059-e350-4394-9e32-0901e740b159/volumes"
Oct 30 22:27:14.251041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:14.251003    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4" event={"ID":"ec9ce949-0e67-4b71-8977-94cd796853cb","Type":"ContainerStarted","Data":"f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238"}
Oct 30 22:27:14.251208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:14.251062    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4" event={"ID":"ec9ce949-0e67-4b71-8977-94cd796853cb","Type":"ContainerStarted","Data":"e39ea75b80e21c1f947ec4ce89da64ce146e28d19a7533ce962f6eead57beba8"}
Oct 30 22:27:14.251208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:14.251084    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:14.291984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:14.291873    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4" podStartSLOduration=2.291857213 podStartE2EDuration="2.291857213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:14.285302313 +0000 UTC m=+1218.324241198" watchObservedRunningTime="2026-10-30 22:27:14.291857213 +0000 UTC m=+1218.330796136"
Oct 30 22:27:15.030800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.030763    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"]
Oct 30 22:27:15.041977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.041944    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ca336c-lnt9f"]
Oct 30 22:27:15.415741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.415708    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"]
Oct 30 22:27:15.421722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.421696    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:15.434267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.434240    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-6f2eba\""
Oct 30 22:27:15.441323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.441293    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"]
Oct 30 22:27:15.480481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.480447    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gjrk\" (UniqueName: \"kubernetes.io/projected/3e043128-f90a-469a-ae0c-a09a7e9a0654-kube-api-access-4gjrk\") pod \"payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj\" (UID: \"3e043128-f90a-469a-ae0c-a09a7e9a0654\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:15.480654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.480573    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e043128-f90a-469a-ae0c-a09a7e9a0654-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj\" (UID: \"3e043128-f90a-469a-ae0c-a09a7e9a0654\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:15.627249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.604129    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gjrk\" (UniqueName: \"kubernetes.io/projected/3e043128-f90a-469a-ae0c-a09a7e9a0654-kube-api-access-4gjrk\") pod \"payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj\" (UID: \"3e043128-f90a-469a-ae0c-a09a7e9a0654\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:15.709354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.676210    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e043128-f90a-469a-ae0c-a09a7e9a0654-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj\" (UID: \"3e043128-f90a-469a-ae0c-a09a7e9a0654\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:15.711824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.692994    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e043128-f90a-469a-ae0c-a09a7e9a0654-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj\" (UID: \"3e043128-f90a-469a-ae0c-a09a7e9a0654\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:15.746192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:15.745924    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gjrk\" (UniqueName: \"kubernetes.io/projected/3e043128-f90a-469a-ae0c-a09a7e9a0654-kube-api-access-4gjrk\") pod \"payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj\" (UID: \"3e043128-f90a-469a-ae0c-a09a7e9a0654\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:16.038564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:16.038461    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:16.104451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:16.104405    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aadb9d28-ad61-4523-b3b2-9bfb95125858" path="/var/lib/kubelet/pods/aadb9d28-ad61-4523-b3b2-9bfb95125858/volumes"
Oct 30 22:27:16.368920 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:27:16.368887    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e043128_f90a_469a_ae0c_a09a7e9a0654.slice/crio-60f0d870aca14ed9e7fc16e8273c96cd0534329962d110153545b7015f3f1da3 WatchSource:0}: Error finding container 60f0d870aca14ed9e7fc16e8273c96cd0534329962d110153545b7015f3f1da3: Status 404 returned error can't find the container with id 60f0d870aca14ed9e7fc16e8273c96cd0534329962d110153545b7015f3f1da3
Oct 30 22:27:16.369843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:16.369820    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"]
Oct 30 22:27:17.051358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:17.051318    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"]
Oct 30 22:27:17.060480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:17.060448    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406f4171-5vzxf"]
Oct 30 22:27:17.354451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:17.354413    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj" event={"ID":"3e043128-f90a-469a-ae0c-a09a7e9a0654","Type":"ContainerStarted","Data":"b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988"}
Oct 30 22:27:17.354867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:17.354486    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:17.354867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:17.354510    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj" event={"ID":"3e043128-f90a-469a-ae0c-a09a7e9a0654","Type":"ContainerStarted","Data":"60f0d870aca14ed9e7fc16e8273c96cd0534329962d110153545b7015f3f1da3"}
Oct 30 22:27:17.392737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:17.392678    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj" podStartSLOduration=2.392662859 podStartE2EDuration="2.392662859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:17.392397352 +0000 UTC m=+1221.431336237" watchObservedRunningTime="2026-10-30 22:27:17.392662859 +0000 UTC m=+1221.431601745"
Oct 30 22:27:18.100035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:18.099995    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77bc59c-a389-4d9f-8092-dc0ec4953902" path="/var/lib/kubelet/pods/e77bc59c-a389-4d9f-8092-dc0ec4953902/volumes"
Oct 30 22:27:18.942484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:18.942443    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"]
Oct 30 22:27:18.948257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:18.948230    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:18.969521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:18.969486    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"]
Oct 30 22:27:18.970241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:18.970197    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:18.970444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:18.970339    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:18.970444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:18.970366    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:18.970657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:18.970457    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:27:19.079665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.079621    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.079865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.079812    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.079926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.079902    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5gr\" (UniqueName: \"kubernetes.io/projected/072910b5-0de0-4afd-a797-73f83f6afd51-kube-api-access-8p5gr\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.181881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.181838    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.182282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.182260    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.182411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.182383    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5gr\" (UniqueName: \"kubernetes.io/projected/072910b5-0de0-4afd-a797-73f83f6afd51-kube-api-access-8p5gr\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.183007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.182982    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.184659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.184631    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.200269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.200159    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5gr\" (UniqueName: \"kubernetes.io/projected/072910b5-0de0-4afd-a797-73f83f6afd51-kube-api-access-8p5gr\") pod \"maas-api-revoke-sub-69b0ca83-vs2dl\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.264846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.264811    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:19.589674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:19.589645    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"]
Oct 30 22:27:19.590422 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:27:19.590387    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072910b5_0de0_4afd_a797_73f83f6afd51.slice/crio-14a99f0441b0425252922e5bb29ac6f84f94aac289d60efbe07dc0d13ecd9d6b WatchSource:0}: Error finding container 14a99f0441b0425252922e5bb29ac6f84f94aac289d60efbe07dc0d13ecd9d6b: Status 404 returned error can't find the container with id 14a99f0441b0425252922e5bb29ac6f84f94aac289d60efbe07dc0d13ecd9d6b
Oct 30 22:27:20.405778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:20.405739    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl" event={"ID":"072910b5-0de0-4afd-a797-73f83f6afd51","Type":"ContainerStarted","Data":"e7cc11095eb109f88ac6ce9207bbcd1f9ac14555b09358e132bb825c2cdbd8ed"}
Oct 30 22:27:20.406287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:20.405798    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl" event={"ID":"072910b5-0de0-4afd-a797-73f83f6afd51","Type":"ContainerStarted","Data":"14a99f0441b0425252922e5bb29ac6f84f94aac289d60efbe07dc0d13ecd9d6b"}
Oct 30 22:27:20.442701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:20.442640    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl" podStartSLOduration=2.442620651 podStartE2EDuration="2.442620651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:20.441157478 +0000 UTC m=+1224.480096363" watchObservedRunningTime="2026-10-30 22:27:20.442620651 +0000 UTC m=+1224.481559537"
Oct 30 22:27:21.445578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:21.445519    4581 generic.go:356] "Generic (PLEG): container finished" podID="072910b5-0de0-4afd-a797-73f83f6afd51" containerID="e7cc11095eb109f88ac6ce9207bbcd1f9ac14555b09358e132bb825c2cdbd8ed" exitCode=0
Oct 30 22:27:21.446029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:21.445647    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl" event={"ID":"072910b5-0de0-4afd-a797-73f83f6afd51","Type":"ContainerDied","Data":"e7cc11095eb109f88ac6ce9207bbcd1f9ac14555b09358e132bb825c2cdbd8ed"}
Oct 30 22:27:22.659944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.659918    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:22.742767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.742673    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072910b5-0de0-4afd-a797-73f83f6afd51-kube-api-access-8p5gr\" (UniqueName: \"kubernetes.io/projected/072910b5-0de0-4afd-a797-73f83f6afd51-kube-api-access-8p5gr\") pod \"072910b5-0de0-4afd-a797-73f83f6afd51\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") "
Oct 30 22:27:22.742767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.742738    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-xks-ca\") pod \"072910b5-0de0-4afd-a797-73f83f6afd51\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") "
Oct 30 22:27:22.743117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.743051    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-openshift-ca\") pod \"072910b5-0de0-4afd-a797-73f83f6afd51\" (UID: \"072910b5-0de0-4afd-a797-73f83f6afd51\") "
Oct 30 22:27:22.743500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.743463    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-openshift-ca" pod "072910b5-0de0-4afd-a797-73f83f6afd51" (UID: "072910b5-0de0-4afd-a797-73f83f6afd51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:22.743771 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.743752    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:22.745105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.745079    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-xks-ca" pod "072910b5-0de0-4afd-a797-73f83f6afd51" (UID: "072910b5-0de0-4afd-a797-73f83f6afd51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:22.745337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.745311    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072910b5-0de0-4afd-a797-73f83f6afd51-kube-api-access-8p5gr" pod "072910b5-0de0-4afd-a797-73f83f6afd51" (UID: "072910b5-0de0-4afd-a797-73f83f6afd51"). InnerVolumeSpecName "kube-api-access-8p5gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:22.845878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.845840    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5gr\" (UniqueName: \"kubernetes.io/projected/072910b5-0de0-4afd-a797-73f83f6afd51-kube-api-access-8p5gr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:22.845878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:22.845874    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072910b5-0de0-4afd-a797-73f83f6afd51-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:23.469189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:23.469156    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"
Oct 30 22:27:23.469189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:23.469174    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl" event={"ID":"072910b5-0de0-4afd-a797-73f83f6afd51","Type":"ContainerDied","Data":"14a99f0441b0425252922e5bb29ac6f84f94aac289d60efbe07dc0d13ecd9d6b"}
Oct 30 22:27:23.469510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:23.469246    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14a99f0441b0425252922e5bb29ac6f84f94aac289d60efbe07dc0d13ecd9d6b"
Oct 30 22:27:25.333929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:25.333896    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:27:28.370730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:28.370693    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 22:27:33.475157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:33.475109    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:27:33.499917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:33.499859    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:33.804550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:33.803340    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:33.877369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:33.877318    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:33.878044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:33.878010    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:27:33.878545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:33.878519    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:33.998948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:33.998891    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.077461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.077343    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.085422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.085359    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"]
Oct 30 22:27:34.182947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.182899    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9955387-thfj8"]
Oct 30 22:27:34.329403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.329293    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:27:34.502383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.502337    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.503630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.503602    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:27:34.769261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.744049    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.769261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.746015    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.815245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.806360    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.839295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.816064    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.839295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.816552    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:27:34.924307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.890350    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.924307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.890764    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:27:34.924307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.891061    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:34.924307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.891757    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:27:34.924307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.891834    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:27:34.924307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:34.891872    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:27:35.126756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.091433    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:35.126756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.122712    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:27:35.183841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.099737    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:27:35.187471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.187438    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:27:35.343546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.343506    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:27:35.345609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.344342    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:27:35.461392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.461185    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:27:35.465365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.465339    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:27:35.528259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.520332    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:35.614285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.614251    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"]
Oct 30 22:27:35.638528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.635490    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"]
Oct 30 22:27:35.710488 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.710409    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b903b2f4-lvhzw"]
Oct 30 22:27:35.796301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.774434    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74dcc53-lhkht"]
Oct 30 22:27:35.810681 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.809594    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:35.812706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.811204    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:27:35.892072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.891345    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:27:35.892072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.891642    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:27:35.953258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:35.949924    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:27:36.116299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.112371    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:27:36.129678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.129632    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:36.383371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.383276    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22838863-375b-409f-a5b6-e230d45a0e2c" path="/var/lib/kubelet/pods/22838863-375b-409f-a5b6-e230d45a0e2c/volumes"
Oct 30 22:27:36.384348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.384323    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d" path="/var/lib/kubelet/pods/4177c778-89ea-4eb8-8c4b-c8aa9fd20d8d/volumes"
Oct 30 22:27:36.387715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.387531    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1eb44e0-1a6c-485b-8d3f-ac110d2b5472" path="/var/lib/kubelet/pods/c1eb44e0-1a6c-485b-8d3f-ac110d2b5472/volumes"
Oct 30 22:27:36.388389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.388372    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:27:36.388555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.388544    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:27:36.388689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.388679    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:27:36.388808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.388799    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:27:36.416056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.416025    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:27:36.421468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.421445    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:27:36.522723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.522691    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"]
Oct 30 22:27:36.524134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.524107    4581 cpu_manager.go:414] "containerMap: removing container" podUID="072910b5-0de0-4afd-a797-73f83f6afd51" containerName="revoke-keys"
Oct 30 22:27:36.524328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.524315    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072910b5-0de0-4afd-a797-73f83f6afd51" containerName="revoke-keys"
Oct 30 22:27:36.524777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.524762    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="072910b5-0de0-4afd-a797-73f83f6afd51" containerName="revoke-keys"
Oct 30 22:27:36.531376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.530845    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.542793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.542750    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"]
Oct 30 22:27:36.543950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.543923    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:36.544104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.544011    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:36.544181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.544109    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:36.544293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.544267    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:27:36.615350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.615256    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.615555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.615438    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.615630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.615550    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vns5d\" (UniqueName: \"kubernetes.io/projected/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-kube-api-access-vns5d\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.717925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.717847    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vns5d\" (UniqueName: \"kubernetes.io/projected/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-kube-api-access-vns5d\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.718123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.718111    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.718193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.718178    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.718983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.718952    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.723089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.722802    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.744018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.743981    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vns5d\" (UniqueName: \"kubernetes.io/projected/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-kube-api-access-vns5d\") pod \"maas-api-revoke-sub-6935ede8-9cx5m\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:36.900524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:36.900484    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:37.249803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:37.249720    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"]
Oct 30 22:27:37.250037 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:27:37.250012    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df9bd00_b4c0_4ecc_bb2d_818d3fd1eea9.slice/crio-028a16a5cddede6ecf091fc24ca7d47be71dcbdc04f02964297940a9c9550916 WatchSource:0}: Error finding container 028a16a5cddede6ecf091fc24ca7d47be71dcbdc04f02964297940a9c9550916: Status 404 returned error can't find the container with id 028a16a5cddede6ecf091fc24ca7d47be71dcbdc04f02964297940a9c9550916
Oct 30 22:27:37.404891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:37.400919    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m" event={"ID":"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9","Type":"ContainerStarted","Data":"028a16a5cddede6ecf091fc24ca7d47be71dcbdc04f02964297940a9c9550916"}
Oct 30 22:27:38.430530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:38.430485    4581 generic.go:356] "Generic (PLEG): container finished" podID="8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9" containerID="8c04472680aec1307149f6ba963ad16021266c4e54e5d37e3d192e2575a07110" exitCode=0
Oct 30 22:27:38.430967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:38.430570    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m" event={"ID":"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9","Type":"ContainerDied","Data":"8c04472680aec1307149f6ba963ad16021266c4e54e5d37e3d192e2575a07110"}
Oct 30 22:27:39.482427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.482385    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:27:39.482897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.482455    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:39.747562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.747523    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:39.816482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.816436    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-xks-ca\") pod \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") "
Oct 30 22:27:39.816753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.816728    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-kube-api-access-vns5d\" (UniqueName: \"kubernetes.io/projected/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-kube-api-access-vns5d\") pod \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") "
Oct 30 22:27:39.817006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.816980    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-openshift-ca\") pod \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\" (UID: \"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9\") "
Oct 30 22:27:39.817564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.817529    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-openshift-ca" pod "8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9" (UID: "8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:39.817922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.817896    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:39.819622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.819587    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-kube-api-access-vns5d" pod "8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9" (UID: "8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9"). InnerVolumeSpecName "kube-api-access-vns5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:39.821388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.821361    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-xks-ca" pod "8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9" (UID: "8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:39.920205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.920164    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:39.920205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:39.920210    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vns5d\" (UniqueName: \"kubernetes.io/projected/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9-kube-api-access-vns5d\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:40.477668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:40.477568    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m" event={"ID":"8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9","Type":"ContainerDied","Data":"028a16a5cddede6ecf091fc24ca7d47be71dcbdc04f02964297940a9c9550916"}
Oct 30 22:27:40.477974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:40.477955    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="028a16a5cddede6ecf091fc24ca7d47be71dcbdc04f02964297940a9c9550916"
Oct 30 22:27:40.478242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:40.478203    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"
Oct 30 22:27:40.833901 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:27:40.833769    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:27:42.533130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:42.533051    4581 generic.go:356] "Generic (PLEG): container finished" podID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerID="70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5" exitCode=0
Oct 30 22:27:42.533130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:42.533122    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" event={"ID":"043e7603-d47f-43c3-aee4-a62b9c1922d2","Type":"ContainerDied","Data":"70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5"}
Oct 30 22:27:42.533720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:42.533701    4581 scope.go:122] "RemoveContainer" containerID="70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5"
Oct 30 22:27:42.535954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:42.535911    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:42.536091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:42.536009    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:42.536091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:42.536061    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:27:43.579242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:43.578959    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" event={"ID":"043e7603-d47f-43c3-aee4-a62b9c1922d2","Type":"ContainerStarted","Data":"0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54"}
Oct 30 22:27:43.736953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:43.736909    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:27:43.737302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:43.737279    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:27:43.737360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:43.737333    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:27:44.737152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:44.737114    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:27:44.737601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:44.737177    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:27:45.737246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:45.737194    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:27:45.737806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:45.737271    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:27:46.034288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:46.034119    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"]
Oct 30 22:27:46.046234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:46.046177    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5c5900-xrm5k"]
Oct 30 22:27:46.102750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:46.102717    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3" path="/var/lib/kubelet/pods/0bcfeb7a-baee-4ec8-a1aa-4844ce7421e3/volumes"
Oct 30 22:27:46.737941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:46.737902    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:27:46.738463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:46.737976    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:27:47.737320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:47.737277    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:27:47.737516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:47.737343    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:27:48.737573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:48.737535    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:27:48.738017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:48.737594    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:27:49.038933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:49.038827    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"]
Oct 30 22:27:49.051745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:49.051708    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f824c259-q6c7f"]
Oct 30 22:27:49.063328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:49.063285    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"]
Oct 30 22:27:49.066868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:49.066828    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0281e42-n5kjm"]
Oct 30 22:27:49.742775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:49.742745    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:27:49.743290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:49.743142    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:27:49.743885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:49.743867    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 22:27:50.102132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:50.102045    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc66158-5167-46a8-8191-a0f3ea99d8a9" path="/var/lib/kubelet/pods/2fc66158-5167-46a8-8191-a0f3ea99d8a9/volumes"
Oct 30 22:27:50.102752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:50.102730    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91279279-1fb9-49d7-bc82-b1519770631b" path="/var/lib/kubelet/pods/91279279-1fb9-49d7-bc82-b1519770631b/volumes"
Oct 30 22:27:54.030299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:54.030264    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"]
Oct 30 22:27:54.042395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:54.042359    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c720e262-kh4wl"]
Oct 30 22:27:54.100164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:54.100127    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417c9962-267d-42b2-a881-2806ae57b9cc" path="/var/lib/kubelet/pods/417c9962-267d-42b2-a881-2806ae57b9cc/volumes"
Oct 30 22:27:54.425793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:54.425756    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"]
Oct 30 22:27:54.426038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:54.426012    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy" containerID="cri-o://6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85" gracePeriod=30
Oct 30 22:27:57.032211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:57.032172    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"]
Oct 30 22:27:57.041172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:57.041142    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598f4cc7-r2k7n"]
Oct 30 22:27:57.650761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:57.650718    4581 scope.go:122] "RemoveContainer" containerID="bf11b0edcdf44febfa2162b6c5840a846fcc3da2f49e5a0a6369a3ccf6f903ca"
Oct 30 22:27:57.665735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:57.665707    4581 scope.go:122] "RemoveContainer" containerID="b99620063d7009e73bdc2a37905b440a764cb939277bf0ea527825e4420ebd31"
Oct 30 22:27:57.678274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:57.678203    4581 scope.go:122] "RemoveContainer" containerID="95a641b67198021cc58b6c73842f76fb393159369d1729646350e61f2cf9ae00"
Oct 30 22:27:57.691573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:57.691544    4581 scope.go:122] "RemoveContainer" containerID="6c5320f783df0e9b417ec9cbdfcd4cf190bd59f10f62b3ea62b15092db7b97ac"
Oct 30 22:27:57.703184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:57.703162    4581 scope.go:122] "RemoveContainer" containerID="442426bad658a261439e02d7c6b247ebfaa469e5891ce0ca21722941d30182de"
Oct 30 22:27:57.715116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:57.715092    4581 scope.go:122] "RemoveContainer" containerID="566c99d089d40d6686cfc4fb606464ba23d6c7854e559fd952cec017b16c2c06"
Oct 30 22:27:58.048675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:58.048593    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"]
Oct 30 22:27:58.061262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:58.061182    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b80472-479lq"]
Oct 30 22:27:58.117000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:58.116963    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71dc3843-fab9-4abb-b27f-b3d96a72f08e" path="/var/lib/kubelet/pods/71dc3843-fab9-4abb-b27f-b3d96a72f08e/volumes"
Oct 30 22:27:58.117698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:58.117658    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ea692e-aea3-499e-b747-f72d9d901aaf" path="/var/lib/kubelet/pods/e0ea692e-aea3-499e-b747-f72d9d901aaf/volumes"
Oct 30 22:27:59.743986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.743956    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:27:59.773202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.773168    4581 generic.go:356] "Generic (PLEG): container finished" podID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerID="6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85" exitCode=0
Oct 30 22:27:59.773402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.773259    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" event={"ID":"8f36daa6-0941-48ff-9e00-105f6eb2eeed","Type":"ContainerDied","Data":"6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85"}
Oct 30 22:27:59.773402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.773327    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h" event={"ID":"8f36daa6-0941-48ff-9e00-105f6eb2eeed","Type":"ContainerDied","Data":"53c8607e1780659e9ed860bb8d8f661248e884998abb400e20c2e61e34a4eab5"}
Oct 30 22:27:59.773402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.773268    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"
Oct 30 22:27:59.773568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.773358    4581 scope.go:122] "RemoveContainer" containerID="6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85"
Oct 30 22:27:59.785959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.785938    4581 scope.go:122] "RemoveContainer" containerID="6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85"
Oct 30 22:27:59.786346 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:27:59.786316    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85\": container with ID starting with 6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85 not found: ID does not exist" containerID="6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85"
Oct 30 22:27:59.786463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.786355    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85"} err="failed to get container status \"6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85\": rpc error: code = NotFound desc = could not find container \"6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85\": container with ID starting with 6deef4a89061745e7962421c94260f4597c5166210e87bc86a605adf20aacb85 not found: ID does not exist"
Oct 30 22:27:59.898278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898247    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-podinfo\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.898531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898430    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-data\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.898602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898553    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-certs\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.898660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898625    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-token\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-token\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.898721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898670    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-socket\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.898830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898799    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istiod-ca-cert\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.898830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898803    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-data" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:59.898988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898847    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-certs" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:59.898988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898890    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-kube-api-access-tz59n\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-kube-api-access-tz59n\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.898988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898931    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-credential-socket\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.898988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.898959    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-envoy\") pod \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\" (UID: \"8f36daa6-0941-48ff-9e00-105f6eb2eeed\") "
Oct 30 22:27:59.899971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.899079    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-socket" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:59.899971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.899549    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istiod-ca-cert" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:59.899971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.899636    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-credential-socket" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:59.900654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.900626    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:59.900743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.900659    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:59.900743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.900674    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:59.900743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.900688    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:59.900743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.900702    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:27:59.902360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.902322    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-kube-api-access-tz59n" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "kube-api-access-tz59n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:59.903643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.903602    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-token" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:59.905564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.905528    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-podinfo" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:59.906053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:27:59.906017    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-envoy" pod "8f36daa6-0941-48ff-9e00-105f6eb2eeed" (UID: "8f36daa6-0941-48ff-9e00-105f6eb2eeed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:00.002772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:00.002686    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:00.002772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:00.002723    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:00.002772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:00.002737    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:00.002772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:00.002751    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz59n\" (UniqueName: \"kubernetes.io/projected/8f36daa6-0941-48ff-9e00-105f6eb2eeed-kube-api-access-tz59n\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:00.123380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:00.123347    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"]
Oct 30 22:28:00.128640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:00.128614    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-l4c4h"]
Oct 30 22:28:02.099210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:02.099165    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" path="/var/lib/kubelet/pods/8f36daa6-0941-48ff-9e00-105f6eb2eeed/volumes"
Oct 30 22:28:04.037783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:04.037746    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"]
Oct 30 22:28:04.046887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:04.046853    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae9919f-n9hv8"]
Oct 30 22:28:04.100874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:04.100836    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f455c4b-b4ea-4af4-8076-a900a6627523" path="/var/lib/kubelet/pods/7f455c4b-b4ea-4af4-8076-a900a6627523/volumes"
Oct 30 22:28:07.044247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:07.043173    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"]
Oct 30 22:28:07.072245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:07.072001    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86019a1-klp2b"]
Oct 30 22:28:08.129043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.129005    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973ff3e1-f903-4c36-ab13-4d5a2057019d" path="/var/lib/kubelet/pods/973ff3e1-f903-4c36-ab13-4d5a2057019d/volumes"
Oct 30 22:28:08.135477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.133617    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:08.171211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.170591    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:08.391478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.391379    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:08.421192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.420258    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:28:08.421192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.420830    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:28:08.457355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.457309    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:08.462334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.462292    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:08.487362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.487311    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:08.715297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.714967    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:08.901245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.893994    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:08.960838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:08.960793    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:28:09.036720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.036392    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 22:28:09.058549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.058290    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:09.135370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.135320    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:09.136274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.136253    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:28:09.186622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.184647    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:28:09.186622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.185187    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:28:09.423252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.401957    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:09.428250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.424685    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:28:09.506247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.479322    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:09.506247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.479997    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:28:09.506247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.486297    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:09.506247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.495677    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:28:09.506247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.496576    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:09.506247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.497085    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:28:09.506247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.501832    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:28:09.563598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.552837    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:28:09.639387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.639339    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 22:28:09.719457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.719347    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:09.737519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.737481    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:28:09.793511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.792601    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 30 22:28:10.013309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.965726    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:28:10.013309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:09.967102    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:28:10.294464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.294359    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:10.492243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.490334    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:10.526290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.525572    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:28:10.554312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.551336    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:10.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.561396    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:10.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.562689    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:28:10.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.570980    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:28:10.614251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.598430    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:28:10.725536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.725488    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:10.863405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:10.863357    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:11.015418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.015365    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:28:11.170301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.163609    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:28:11.408430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.408375    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:11.563953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.547701    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:11.563953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.548889    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:11.577284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.564535    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:11.577284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.564932    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:28:11.606264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.578084    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:28:11.606264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.580451    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:11.632245 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:28:11.607444    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:28:11.632245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.610005    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:28:11.650942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.649872    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:11.731528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.731481    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:11.824699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:11.824601    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded"
Oct 30 22:28:12.044421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.043338    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:12.044421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.043970    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:28:12.208266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.182355    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:12.408247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.383064    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:28:12.481241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.481113    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:28:12.560448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.560403    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:28:12.574729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.574688    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:12.575697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.575663    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:28:12.576124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.576103    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:28:12.601688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.585109    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:12.614246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.610246    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:12.637077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.637031    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:12.687901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.687844    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:28:12.744790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.744675    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:28:12.902342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.902274    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"]
Oct 30 22:28:12.969575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:12.969450    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"]
Oct 30 22:28:13.069430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.052951    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:13.071473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.071432    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e3ec73-cqkpz"]
Oct 30 22:28:13.132715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.130728    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-119a68a7-qtrw6"]
Oct 30 22:28:13.435457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.429767    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:28:13.514429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.514365    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": context deadline exceeded"
Oct 30 22:28:13.577410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.577354    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:13.578944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.578913    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:28:13.595253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.593320    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:13.603574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.603532    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:28:13.605112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.604976    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 22:28:13.622299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.622257    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:13.696065 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.695956    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:28:13.732684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.732652    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:28:13.742723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.738506    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:28:13.746267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:13.746207    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:14.091364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.055552    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:28:14.189976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.189935    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:28:14.278247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.249982    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:28:14.337287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.337250    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:28:14.430182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.430139    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:14.614240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.611726    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:14.623507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.623465    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:14.628645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.628605    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3104be-1fbe-496f-b457-506d6b4f39fc" path="/var/lib/kubelet/pods/9e3104be-1fbe-496f-b457-506d6b4f39fc/volumes"
Oct 30 22:28:14.629588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.629561    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1420a2-f6d7-466d-b91f-1a8b6171ebd1" path="/var/lib/kubelet/pods/ef1420a2-f6d7-466d-b91f-1a8b6171ebd1/volumes"
Oct 30 22:28:14.630249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.630197    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:28:14.715299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.693573    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:28:14.715299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.715253    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:28:14.716997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.716976    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:28:14.717316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.717204    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:28:14.717513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.717420    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:14.741510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:14.741481    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:28:15.685288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.684307    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"]
Oct 30 22:28:15.687164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.687134    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9" containerName="revoke-keys"
Oct 30 22:28:15.687357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.687340    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9" containerName="revoke-keys"
Oct 30 22:28:15.687581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.687569    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy"
Oct 30 22:28:15.687720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.687707    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy"
Oct 30 22:28:15.688081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.688067    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f36daa6-0941-48ff-9e00-105f6eb2eeed" containerName="istio-proxy"
Oct 30 22:28:15.688230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.688203    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9" containerName="revoke-keys"
Oct 30 22:28:15.700350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.700317    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.715103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.714330    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-510f80-openshift-default-dockercfg-w2m4c\""
Oct 30 22:28:15.728497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.728413    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"]
Oct 30 22:28:15.833631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.833600    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.833796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.833756    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.833861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.833802    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.833999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.833974    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.834073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.834023    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.834125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.834073    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.834248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.834199    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.834411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.834343    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh42t\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-kube-api-access-fh42t\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.834481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.834441    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.973246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.973133    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.973526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.973503    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.974040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.974010    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istiod-ca-cert\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.974178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.973924    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.974703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.974321    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.974861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.974844    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.975031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.975014    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.975156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.975141    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.975377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.975357    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh42t\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-kube-api-access-fh42t\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.975580 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.975563    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.975790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.975773    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.976413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.976392    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-certs\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.976781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.976760    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-credential-socket\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.977870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.977847    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-data\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:15.996698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:15.996668    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-envoy\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:16.006923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:16.006848    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-token\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:16.021970 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:16.021860    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-podinfo\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:16.107095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:16.106913    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh42t\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-kube-api-access-fh42t\") pod \"e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") " pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:16.334514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:16.334422    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:16.705094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:16.705063    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"]
Oct 30 22:28:16.705667 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:28:16.705642    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0e492e_4281_4be3_bed8_8f57b2f13eb5.slice/crio-5ca8309fd3e5bb72896c9088ee7896cecac6b57fe625565b2035c1b9deb37aad WatchSource:0}: Error finding container 5ca8309fd3e5bb72896c9088ee7896cecac6b57fe625565b2035c1b9deb37aad: Status 404 returned error can't find the container with id 5ca8309fd3e5bb72896c9088ee7896cecac6b57fe625565b2035c1b9deb37aad
Oct 30 22:28:16.709189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:16.709155    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:16.709293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:16.709276    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:16.709335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:16.709320    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:17.036569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:17.036461    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"]
Oct 30 22:28:17.045627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:17.045588    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c137beb-t49ck"]
Oct 30 22:28:17.476304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:17.476259    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" event={"ID":"9c0e492e-4281-4be3-bed8-8f57b2f13eb5","Type":"ContainerStarted","Data":"ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be"}
Oct 30 22:28:17.476558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:17.476335    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" event={"ID":"9c0e492e-4281-4be3-bed8-8f57b2f13eb5","Type":"ContainerStarted","Data":"5ca8309fd3e5bb72896c9088ee7896cecac6b57fe625565b2035c1b9deb37aad"}
Oct 30 22:28:17.525829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:17.525751    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podStartSLOduration=2.525731182 podStartE2EDuration="2.525731182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:17.520017482 +0000 UTC m=+1281.558956369" watchObservedRunningTime="2026-10-30 22:28:17.525731182 +0000 UTC m=+1281.564670069"
Oct 30 22:28:18.031547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:18.031508    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"]
Oct 30 22:28:18.040804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:18.040766    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1749884-wh7dz"]
Oct 30 22:28:18.106909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:18.106869    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6840a624-d9f5-4dbe-87f7-37c5df9e5e35" path="/var/lib/kubelet/pods/6840a624-d9f5-4dbe-87f7-37c5df9e5e35/volumes"
Oct 30 22:28:18.107684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:18.107651    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21816b3-807d-496b-b707-58dd440613de" path="/var/lib/kubelet/pods/a21816b3-807d-496b-b707-58dd440613de/volumes"
Oct 30 22:28:18.338122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:18.338039    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:18.346161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:18.346129    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:18.489432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:18.489351    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:18.490591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:18.490567    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:28:22.267587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.265939    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"]
Oct 30 22:28:22.267587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.265996    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268846d-c9spq"]
Oct 30 22:28:22.339391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.339351    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"]
Oct 30 22:28:22.423496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.423455    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"]
Oct 30 22:28:22.423694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.423622    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.436041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.436011    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:22.436262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.436191    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:22.447797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.447760    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:22.447960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.447854    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:28:22.599538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.599435    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.599736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.599644    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/1a3fbf76-da83-476a-89c2-3cac5abee640-kube-api-access-szwhq\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.599856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.599835    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.702813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.702756    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.703070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.703049    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.703203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.703178    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/1a3fbf76-da83-476a-89c2-3cac5abee640-kube-api-access-szwhq\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.704132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.704096    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.706914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.706885    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.721474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.721431    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/1a3fbf76-da83-476a-89c2-3cac5abee640-kube-api-access-szwhq\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-wfsw5\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:22.753292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:22.753245    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:23.207039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:23.086529    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"]
Oct 30 22:28:23.584211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:23.584170    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5" event={"ID":"1a3fbf76-da83-476a-89c2-3cac5abee640","Type":"ContainerStarted","Data":"a3f7eba88a0abc01994e5ab1de47e6f6474075aaeef29b52c917a11ffde9b2fd"}
Oct 30 22:28:23.584694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:23.584287    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5" event={"ID":"1a3fbf76-da83-476a-89c2-3cac5abee640","Type":"ContainerStarted","Data":"5ee1740cb6fb9aa140dac5f31867a11cfbf318cefe6ca3ac983b99037cb02c2e"}
Oct 30 22:28:23.617839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:23.617781    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5" podStartSLOduration=1.617763646 podStartE2EDuration="1.617763646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:23.616681308 +0000 UTC m=+1287.655620194" watchObservedRunningTime="2026-10-30 22:28:23.617763646 +0000 UTC m=+1287.656702532"
Oct 30 22:28:24.116534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.116497    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a189dbde-a4e1-40f6-90a5-745153598f1d" path="/var/lib/kubelet/pods/a189dbde-a4e1-40f6-90a5-745153598f1d/volumes"
Oct 30 22:28:24.135970 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.135941    4581 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:28:24.136129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.136000    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:28:24.596003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.595964    4581 generic.go:356] "Generic (PLEG): container finished" podID="1a3fbf76-da83-476a-89c2-3cac5abee640" containerID="a3f7eba88a0abc01994e5ab1de47e6f6474075aaeef29b52c917a11ffde9b2fd" exitCode=0
Oct 30 22:28:24.596567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.596058    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5" event={"ID":"1a3fbf76-da83-476a-89c2-3cac5abee640","Type":"ContainerDied","Data":"a3f7eba88a0abc01994e5ab1de47e6f6474075aaeef29b52c917a11ffde9b2fd"}
Oct 30 22:28:24.598368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.598344    4581 generic.go:356] "Generic (PLEG): container finished" podID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerID="e7480b2d393ec706c625bb4f3e72c7b51aef309dd99879cce3531c91ecd09408" exitCode=0
Oct 30 22:28:24.598495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.598416    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" event={"ID":"043d11d3-326a-4c67-817a-7b9a7cdf9596","Type":"ContainerDied","Data":"e7480b2d393ec706c625bb4f3e72c7b51aef309dd99879cce3531c91ecd09408"}
Oct 30 22:28:24.598956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.598934    4581 scope.go:122] "RemoveContainer" containerID="e7480b2d393ec706c625bb4f3e72c7b51aef309dd99879cce3531c91ecd09408"
Oct 30 22:28:24.600762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.600738    4581 generic.go:356] "Generic (PLEG): container finished" podID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerID="cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5" exitCode=0
Oct 30 22:28:24.601126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.601103    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" event={"ID":"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1","Type":"ContainerDied","Data":"cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5"}
Oct 30 22:28:24.601337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.601302    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:24.601451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.601392    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:24.601451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.601444    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:24.601588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.601570    4581 scope.go:122] "RemoveContainer" containerID="cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5"
Oct 30 22:28:24.603286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.603208    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:24.603370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.603337    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:24.603414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:24.603399    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:28:25.625236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:25.624269    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" event={"ID":"043d11d3-326a-4c67-817a-7b9a7cdf9596","Type":"ContainerStarted","Data":"9f1c65faf8cb75b9e91aef8361bfc402fbba70affc3da950c2badb79b778668c"}
Oct 30 22:28:25.629267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:25.628776    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" event={"ID":"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1","Type":"ContainerStarted","Data":"6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028"}
Oct 30 22:28:25.739248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:25.720028    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:28:25.739544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:25.733661    4581 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:28:25.739718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:25.739689    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:28:26.019509 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.019350    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:28:26.020681 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.020649    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:28:26.020828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.020707    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:28:26.231604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.231523    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:26.338500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.338373    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-xks-ca\") pod \"1a3fbf76-da83-476a-89c2-3cac5abee640\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") "
Oct 30 22:28:26.338500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.338467    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-openshift-ca\") pod \"1a3fbf76-da83-476a-89c2-3cac5abee640\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") "
Oct 30 22:28:26.338500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.338497    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3fbf76-da83-476a-89c2-3cac5abee640-kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/1a3fbf76-da83-476a-89c2-3cac5abee640-kube-api-access-szwhq\") pod \"1a3fbf76-da83-476a-89c2-3cac5abee640\" (UID: \"1a3fbf76-da83-476a-89c2-3cac5abee640\") "
Oct 30 22:28:26.340893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.339620    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-openshift-ca" pod "1a3fbf76-da83-476a-89c2-3cac5abee640" (UID: "1a3fbf76-da83-476a-89c2-3cac5abee640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:26.342636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.342589    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-xks-ca" pod "1a3fbf76-da83-476a-89c2-3cac5abee640" (UID: "1a3fbf76-da83-476a-89c2-3cac5abee640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:26.342779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.342655    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3fbf76-da83-476a-89c2-3cac5abee640-kube-api-access-szwhq" pod "1a3fbf76-da83-476a-89c2-3cac5abee640" (UID: "1a3fbf76-da83-476a-89c2-3cac5abee640"). InnerVolumeSpecName "kube-api-access-szwhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:26.441850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.441819    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:26.442048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.442038    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3fbf76-da83-476a-89c2-3cac5abee640-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:26.442115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.442106    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/1a3fbf76-da83-476a-89c2-3cac5abee640-kube-api-access-szwhq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:26.645483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.645449    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"
Oct 30 22:28:26.647374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.647337    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5" event={"ID":"1a3fbf76-da83-476a-89c2-3cac5abee640","Type":"ContainerDied","Data":"5ee1740cb6fb9aa140dac5f31867a11cfbf318cefe6ca3ac983b99037cb02c2e"}
Oct 30 22:28:26.647508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.647398    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ee1740cb6fb9aa140dac5f31867a11cfbf318cefe6ca3ac983b99037cb02c2e"
Oct 30 22:28:26.703243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.702668    4581 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:28:26.703243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:26.702734    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:28:27.014147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:27.013386    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:28:27.014147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:27.013512    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:28:27.215381 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:27.215345    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:28:27.738244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:27.731942    4581 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:28:27.738244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:27.732037    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:28:28.011302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:28.010747    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:28:28.011302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:28.010815    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:28:28.701557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:28.701523    4581 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:28:28.701913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:28.701878    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:28:29.012302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:29.011651    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:28:29.012302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:29.011716    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:28:29.705733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:29.705697    4581 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:28:29.724243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:29.705993    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:28:30.013493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:30.013386    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:28:30.013493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:30.013453    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:28:30.699249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:30.698665    4581 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:28:30.699249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:30.698761    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:28:31.007708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:31.007609    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:28:31.008016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:31.007986    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:28:31.703898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:31.703863    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:28:31.704417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:31.704249    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:28:31.705101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:31.705079    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:28:32.011365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:32.011287    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:28:32.012620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:32.012599    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 22:28:40.820307 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:28:40.820190    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:28:45.036902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:45.036865    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"]
Oct 30 22:28:45.044498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:45.044469    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334e36d8-r267h"]
Oct 30 22:28:46.101982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:46.101946    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5164dde6-b472-418d-98c3-51cd5b0009b1" path="/var/lib/kubelet/pods/5164dde6-b472-418d-98c3-51cd5b0009b1/volumes"
Oct 30 22:28:53.831658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.831620    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"]
Oct 30 22:28:53.832416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.832400    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1a3fbf76-da83-476a-89c2-3cac5abee640" containerName="revoke-keys"
Oct 30 22:28:53.832499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.832417    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3fbf76-da83-476a-89c2-3cac5abee640" containerName="revoke-keys"
Oct 30 22:28:53.832583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.832572    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3fbf76-da83-476a-89c2-3cac5abee640" containerName="revoke-keys"
Oct 30 22:28:53.835451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.835431    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:53.857257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.857202    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"]
Oct 30 22:28:53.859456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.859431    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:53.859633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.859570    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:28:53.859730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.859663    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:53.859730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.859709    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:53.955793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.955758    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:53.956032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.956017    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:53.956094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:53.956083    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6gxz\" (UniqueName: \"kubernetes.io/projected/55332ab2-75b3-4d97-a454-6f63e404af4d-kube-api-access-z6gxz\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:54.057932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:54.057865    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:54.058230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:54.058193    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:54.058332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:54.058316    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gxz\" (UniqueName: \"kubernetes.io/projected/55332ab2-75b3-4d97-a454-6f63e404af4d-kube-api-access-z6gxz\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:54.058702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:54.058680    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:54.060939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:54.060897    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:54.075773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:54.075741    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gxz\" (UniqueName: \"kubernetes.io/projected/55332ab2-75b3-4d97-a454-6f63e404af4d-kube-api-access-z6gxz\") pod \"maas-api-revoke-sub-b9cd855d-bmx2d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:54.149847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:54.149807    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:54.485822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:54.485785    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"]
Oct 30 22:28:54.486649 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:28:54.486616    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55332ab2_75b3_4d97_a454_6f63e404af4d.slice/crio-7e9de750c3d6301bb849708fad9cce7a879683b3ef52258a37657f726c231128 WatchSource:0}: Error finding container 7e9de750c3d6301bb849708fad9cce7a879683b3ef52258a37657f726c231128: Status 404 returned error can't find the container with id 7e9de750c3d6301bb849708fad9cce7a879683b3ef52258a37657f726c231128
Oct 30 22:28:55.134033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:55.133992    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d" event={"ID":"55332ab2-75b3-4d97-a454-6f63e404af4d","Type":"ContainerStarted","Data":"a850e889d47fabae82c76c34da19e446ff76087c63374bbd759bc359c14f7344"}
Oct 30 22:28:55.134530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:55.134056    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d" event={"ID":"55332ab2-75b3-4d97-a454-6f63e404af4d","Type":"ContainerStarted","Data":"7e9de750c3d6301bb849708fad9cce7a879683b3ef52258a37657f726c231128"}
Oct 30 22:28:55.174714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:55.174642    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d" podStartSLOduration=2.174620897 podStartE2EDuration="2.174620897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:55.166708726 +0000 UTC m=+1319.205647609" watchObservedRunningTime="2026-10-30 22:28:55.174620897 +0000 UTC m=+1319.213559785"
Oct 30 22:28:56.146284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:56.146203    4581 generic.go:356] "Generic (PLEG): container finished" podID="55332ab2-75b3-4d97-a454-6f63e404af4d" containerID="a850e889d47fabae82c76c34da19e446ff76087c63374bbd759bc359c14f7344" exitCode=0
Oct 30 22:28:56.146700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:56.146303    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d" event={"ID":"55332ab2-75b3-4d97-a454-6f63e404af4d","Type":"ContainerDied","Data":"a850e889d47fabae82c76c34da19e446ff76087c63374bbd759bc359c14f7344"}
Oct 30 22:28:57.377093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.377060    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:57.527834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.527737    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55332ab2-75b3-4d97-a454-6f63e404af4d-kube-api-access-z6gxz\" (UniqueName: \"kubernetes.io/projected/55332ab2-75b3-4d97-a454-6f63e404af4d-kube-api-access-z6gxz\") pod \"55332ab2-75b3-4d97-a454-6f63e404af4d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") "
Oct 30 22:28:57.528014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.527967    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-openshift-ca\") pod \"55332ab2-75b3-4d97-a454-6f63e404af4d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") "
Oct 30 22:28:57.528014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.528000    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-xks-ca\") pod \"55332ab2-75b3-4d97-a454-6f63e404af4d\" (UID: \"55332ab2-75b3-4d97-a454-6f63e404af4d\") "
Oct 30 22:28:57.528427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.528400    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-openshift-ca" pod "55332ab2-75b3-4d97-a454-6f63e404af4d" (UID: "55332ab2-75b3-4d97-a454-6f63e404af4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:57.528766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.528712    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:57.530268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.530201    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55332ab2-75b3-4d97-a454-6f63e404af4d-kube-api-access-z6gxz" pod "55332ab2-75b3-4d97-a454-6f63e404af4d" (UID: "55332ab2-75b3-4d97-a454-6f63e404af4d"). InnerVolumeSpecName "kube-api-access-z6gxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:57.530389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.530370    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-xks-ca" pod "55332ab2-75b3-4d97-a454-6f63e404af4d" (UID: "55332ab2-75b3-4d97-a454-6f63e404af4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:57.630559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.630522    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55332ab2-75b3-4d97-a454-6f63e404af4d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:57.630559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.630554    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6gxz\" (UniqueName: \"kubernetes.io/projected/55332ab2-75b3-4d97-a454-6f63e404af4d-kube-api-access-z6gxz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:28:57.798892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.798765    4581 scope.go:122] "RemoveContainer" containerID="5d290744323411cb679e09ff532dc05ccdec6b9e3120cf4dd0c95576b2c329ee"
Oct 30 22:28:57.815013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.814985    4581 scope.go:122] "RemoveContainer" containerID="e9ef0f43c91170e5a025622da72ebabde4c72220a26a035559ba21b2d3cb38f3"
Oct 30 22:28:57.828880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.828856    4581 scope.go:122] "RemoveContainer" containerID="1ecfbb903fbdbbc9db9c47d1df0b0a7f73d5fb8a6bb0cfcc857fcd80d7912b7a"
Oct 30 22:28:57.845262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.845206    4581 scope.go:122] "RemoveContainer" containerID="676bfa4f452877700281e94de7113c088e0e14655e2aec5bd06cff4cc12396ce"
Oct 30 22:28:57.860307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.860283    4581 scope.go:122] "RemoveContainer" containerID="726d6521fade1f657d1941f148aa605401e43c619bab71459740a11d518c1896"
Oct 30 22:28:57.873012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.872986    4581 scope.go:122] "RemoveContainer" containerID="74c6b74f59f3671a442c4e33e920c69e459d8c346bf565fed26ca9f94fd3c9d3"
Oct 30 22:28:57.885746 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.885720    4581 scope.go:122] "RemoveContainer" containerID="487260425dcf18a29326d66e31a4b9a98137408d32b3cdbf661ef981ab2018d0"
Oct 30 22:28:57.900422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.900401    4581 scope.go:122] "RemoveContainer" containerID="6e42aa695b0164b9d9e51d94f4b230c5672af6f51cc8ed900b00885c333981ed"
Oct 30 22:28:57.911852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.911829    4581 scope.go:122] "RemoveContainer" containerID="205ba62da10c74729b61a3e36cf8982d3f8fcc96866da8a09f6c24afd7cd70bd"
Oct 30 22:28:57.923139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.923109    4581 scope.go:122] "RemoveContainer" containerID="9ac8568bac98d94a711a4fc6c418fcd5e1181d92469fe9d8e23a71f7eabe1564"
Oct 30 22:28:57.933824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.933805    4581 scope.go:122] "RemoveContainer" containerID="107ec363cbc2f4d970b80876180ffc2469709f1c07d939f6575e06a191c01c01"
Oct 30 22:28:57.944380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.944359    4581 scope.go:122] "RemoveContainer" containerID="be63fc1e66c418f0a51a4b01fe546158357f311c983ce10e90663f827ab779ce"
Oct 30 22:28:57.954890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.954867    4581 scope.go:122] "RemoveContainer" containerID="fffd2044bdd08286a87bc49f1c26fd8727cf1fb59c5290c785a83f63b5eebc78"
Oct 30 22:28:57.967436 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:57.967408    4581 scope.go:122] "RemoveContainer" containerID="dffc4f813135171b775ad597f26c69d8fb6ca58fbec8ca4935e13ddcaaaf7b4b"
Oct 30 22:28:58.174449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:58.174421    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"
Oct 30 22:28:58.174633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:58.174450    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d" event={"ID":"55332ab2-75b3-4d97-a454-6f63e404af4d","Type":"ContainerDied","Data":"7e9de750c3d6301bb849708fad9cce7a879683b3ef52258a37657f726c231128"}
Oct 30 22:28:58.174633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:28:58.174515    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e9de750c3d6301bb849708fad9cce7a879683b3ef52258a37657f726c231128"
Oct 30 22:29:00.110120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.110085    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"]
Oct 30 22:29:00.111018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.111000    4581 cpu_manager.go:414] "containerMap: removing container" podUID="55332ab2-75b3-4d97-a454-6f63e404af4d" containerName="revoke-keys"
Oct 30 22:29:00.111062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.111020    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55332ab2-75b3-4d97-a454-6f63e404af4d" containerName="revoke-keys"
Oct 30 22:29:00.111151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.111142    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="55332ab2-75b3-4d97-a454-6f63e404af4d" containerName="revoke-keys"
Oct 30 22:29:00.116257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.116230    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.134477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.134449    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"]
Oct 30 22:29:00.138265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.138238    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:00.138265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.138267    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:00.138451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.138287    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:29:00.138451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.138333    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:00.298831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.298793    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.299030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.298956    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.299227 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.299186    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7rc5\" (UniqueName: \"kubernetes.io/projected/4f5ba764-e1df-4205-aab3-2a79e9168fe6-kube-api-access-j7rc5\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.400661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.400576    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rc5\" (UniqueName: \"kubernetes.io/projected/4f5ba764-e1df-4205-aab3-2a79e9168fe6-kube-api-access-j7rc5\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.400825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.400757    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.400881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.400832    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.401541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.401515    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.403280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.403261    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.417174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.417142    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rc5\" (UniqueName: \"kubernetes.io/projected/4f5ba764-e1df-4205-aab3-2a79e9168fe6-kube-api-access-j7rc5\") pod \"maas-api-revoke-sub-295c65aa-786h2\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.430281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.430251    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:00.744337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:00.744295    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"]
Oct 30 22:29:00.747021 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:29:00.746976    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5ba764_e1df_4205_aab3_2a79e9168fe6.slice/crio-47cdd2f92b3aed8c3a09b832e46b57e82a588183f4bf25694f60449dcb4ae4af WatchSource:0}: Error finding container 47cdd2f92b3aed8c3a09b832e46b57e82a588183f4bf25694f60449dcb4ae4af: Status 404 returned error can't find the container with id 47cdd2f92b3aed8c3a09b832e46b57e82a588183f4bf25694f60449dcb4ae4af
Oct 30 22:29:01.210774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:01.210732    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2" event={"ID":"4f5ba764-e1df-4205-aab3-2a79e9168fe6","Type":"ContainerStarted","Data":"2040d1cddd09764cc6aea0a36d318684e26ed8f466fc31a183d8ee5fb4c7ac9b"}
Oct 30 22:29:01.211171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:01.210800    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2" event={"ID":"4f5ba764-e1df-4205-aab3-2a79e9168fe6","Type":"ContainerStarted","Data":"47cdd2f92b3aed8c3a09b832e46b57e82a588183f4bf25694f60449dcb4ae4af"}
Oct 30 22:29:01.250431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:01.250374    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2" podStartSLOduration=1.2503580890000001 podStartE2EDuration="1.250358089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:01.245409269 +0000 UTC m=+1325.284348155" watchObservedRunningTime="2026-10-30 22:29:01.250358089 +0000 UTC m=+1325.289296974"
Oct 30 22:29:02.233480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:02.233443    4581 generic.go:356] "Generic (PLEG): container finished" podID="4f5ba764-e1df-4205-aab3-2a79e9168fe6" containerID="2040d1cddd09764cc6aea0a36d318684e26ed8f466fc31a183d8ee5fb4c7ac9b" exitCode=0
Oct 30 22:29:02.233904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:02.233531    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2" event={"ID":"4f5ba764-e1df-4205-aab3-2a79e9168fe6","Type":"ContainerDied","Data":"2040d1cddd09764cc6aea0a36d318684e26ed8f466fc31a183d8ee5fb4c7ac9b"}
Oct 30 22:29:03.446032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.446004    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:03.551384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.551266    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-openshift-ca\") pod \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") "
Oct 30 22:29:03.551384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.551357    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5ba764-e1df-4205-aab3-2a79e9168fe6-kube-api-access-j7rc5\" (UniqueName: \"kubernetes.io/projected/4f5ba764-e1df-4205-aab3-2a79e9168fe6-kube-api-access-j7rc5\") pod \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") "
Oct 30 22:29:03.551743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.551422    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-xks-ca\") pod \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\" (UID: \"4f5ba764-e1df-4205-aab3-2a79e9168fe6\") "
Oct 30 22:29:03.552148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.552100    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-openshift-ca" pod "4f5ba764-e1df-4205-aab3-2a79e9168fe6" (UID: "4f5ba764-e1df-4205-aab3-2a79e9168fe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:03.555838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.555808    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-xks-ca" pod "4f5ba764-e1df-4205-aab3-2a79e9168fe6" (UID: "4f5ba764-e1df-4205-aab3-2a79e9168fe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:03.556014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.555993    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5ba764-e1df-4205-aab3-2a79e9168fe6-kube-api-access-j7rc5" pod "4f5ba764-e1df-4205-aab3-2a79e9168fe6" (UID: "4f5ba764-e1df-4205-aab3-2a79e9168fe6"). InnerVolumeSpecName "kube-api-access-j7rc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:03.653358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.653316    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:03.653358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.653352    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7rc5\" (UniqueName: \"kubernetes.io/projected/4f5ba764-e1df-4205-aab3-2a79e9168fe6-kube-api-access-j7rc5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:03.653358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:03.653365    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5ba764-e1df-4205-aab3-2a79e9168fe6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:04.262930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:04.262894    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"
Oct 30 22:29:04.263136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:04.262927    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2" event={"ID":"4f5ba764-e1df-4205-aab3-2a79e9168fe6","Type":"ContainerDied","Data":"47cdd2f92b3aed8c3a09b832e46b57e82a588183f4bf25694f60449dcb4ae4af"}
Oct 30 22:29:04.263136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:04.262980    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47cdd2f92b3aed8c3a09b832e46b57e82a588183f4bf25694f60449dcb4ae4af"
Oct 30 22:29:09.148576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.148540    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"]
Oct 30 22:29:09.148998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.148754    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy" containerID="cri-o://7c71a65413e7ba1114212a26e7f33c30c8f8bc262be2b9eae706fb828ec29afe" gracePeriod=30
Oct 30 22:29:09.192020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.191526    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"]
Oct 30 22:29:09.192020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.191781    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy" containerID="cri-o://dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57" gracePeriod=30
Oct 30 22:29:09.226245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.225552    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"]
Oct 30 22:29:09.226245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.225911    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy" containerID="cri-o://762af472bc458080588b7192c41771ae4a4619665e2b6a0db4fa47e764bbcc13" gracePeriod=30
Oct 30 22:29:09.241246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.240659    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"]
Oct 30 22:29:09.241246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.240936    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy" containerID="cri-o://8acce8cd0e073153aa418904524a83822fcc64acf9947645e2051ad66000ea95" gracePeriod=30
Oct 30 22:29:09.279507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.265321    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"]
Oct 30 22:29:09.279507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.265617    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" containerID="cri-o://99b0234912798ff5d5e761f2b8192181ed27f73519263d4f4f4ef6bbaf65ecdb" gracePeriod=30
Oct 30 22:29:09.402066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.401976    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:09.402271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:09.402056    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:10.932339 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:29:10.932297    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:29:14.643519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.639758    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:29:14.643519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.641293    4581 generic.go:356] "Generic (PLEG): container finished" podID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerID="99b0234912798ff5d5e761f2b8192181ed27f73519263d4f4f4ef6bbaf65ecdb" exitCode=0
Oct 30 22:29:14.643519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.641391    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" event={"ID":"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf","Type":"ContainerDied","Data":"99b0234912798ff5d5e761f2b8192181ed27f73519263d4f4f4ef6bbaf65ecdb"}
Oct 30 22:29:14.643519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.643070    4581 generic.go:356] "Generic (PLEG): container finished" podID="8c720270-bcf8-433a-8edc-bb68f329640c" containerID="dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57" exitCode=0
Oct 30 22:29:14.643519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.643353    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" event={"ID":"8c720270-bcf8-433a-8edc-bb68f329640c","Type":"ContainerDied","Data":"dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57"}
Oct 30 22:29:14.643519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.643401    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66" event={"ID":"8c720270-bcf8-433a-8edc-bb68f329640c","Type":"ContainerDied","Data":"b097a57db12ea77ca7436a1a371e092e3868de594e89a8d851b1be379022c2f3"}
Oct 30 22:29:14.643519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.643432    4581 scope.go:122] "RemoveContainer" containerID="dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57"
Oct 30 22:29:14.644303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.643600    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"
Oct 30 22:29:14.648183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.647908    4581 generic.go:356] "Generic (PLEG): container finished" podID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerID="8acce8cd0e073153aa418904524a83822fcc64acf9947645e2051ad66000ea95" exitCode=0
Oct 30 22:29:14.648183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.648012    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" event={"ID":"90b6695b-e9f4-4fd0-8979-d06d71d6a085","Type":"ContainerDied","Data":"8acce8cd0e073153aa418904524a83822fcc64acf9947645e2051ad66000ea95"}
Oct 30 22:29:14.666274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.650521    4581 generic.go:356] "Generic (PLEG): container finished" podID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerID="7c71a65413e7ba1114212a26e7f33c30c8f8bc262be2b9eae706fb828ec29afe" exitCode=0
Oct 30 22:29:14.666274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.650762    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" event={"ID":"741a2f67-6328-40cb-8866-cc4e17b0584d","Type":"ContainerDied","Data":"7c71a65413e7ba1114212a26e7f33c30c8f8bc262be2b9eae706fb828ec29afe"}
Oct 30 22:29:14.666274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.653470    4581 generic.go:356] "Generic (PLEG): container finished" podID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerID="762af472bc458080588b7192c41771ae4a4619665e2b6a0db4fa47e764bbcc13" exitCode=0
Oct 30 22:29:14.666274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.653501    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" event={"ID":"603e680b-f3b7-425b-83ac-b7e7051cd801","Type":"ContainerDied","Data":"762af472bc458080588b7192c41771ae4a4619665e2b6a0db4fa47e764bbcc13"}
Oct 30 22:29:14.666274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.662578    4581 scope.go:122] "RemoveContainer" containerID="dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57"
Oct 30 22:29:14.666274 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:29:14.662949    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57\": container with ID starting with dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57 not found: ID does not exist" containerID="dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57"
Oct 30 22:29:14.666274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.662975    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57"} err="failed to get container status \"dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57\": rpc error: code = NotFound desc = could not find container \"dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57\": container with ID starting with dc2e27bc7f10ef6fe8b62ca4c5faf64e4c0e316e5b8e2fea07dcc856dde34b57 not found: ID does not exist"
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.714458    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-socket\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.714513    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-kube-api-access-wr5kx\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-kube-api-access-wr5kx\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.714621    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-certs\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.714739    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-envoy\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.714794    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c720270-bcf8-433a-8edc-bb68f329640c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c720270-bcf8-433a-8edc-bb68f329640c-istiod-ca-cert\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.714846    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-data\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.714876    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c720270-bcf8-433a-8edc-bb68f329640c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c720270-bcf8-433a-8edc-bb68f329640c-istio-podinfo\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.715108    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-credential-socket\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.715230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.715151    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-istio-token\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-istio-token\") pod \"8c720270-bcf8-433a-8edc-bb68f329640c\" (UID: \"8c720270-bcf8-433a-8edc-bb68f329640c\") "
Oct 30 22:29:14.718709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.718431    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-credential-socket" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.719684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.719229    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c720270-bcf8-433a-8edc-bb68f329640c-istiod-ca-cert" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.720156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.719946    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-data" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.720645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.720327    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-socket" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.720863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.720807    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-certs" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.722279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.721367    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.722279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.721391    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.722279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.721407    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.722279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.721420    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c720270-bcf8-433a-8edc-bb68f329640c-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.722279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.721432    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.735253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.731685    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-envoy" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.735253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.734513    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c720270-bcf8-433a-8edc-bb68f329640c-istio-podinfo" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.735511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.735480    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-kube-api-access-wr5kx" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "kube-api-access-wr5kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.735652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.735617    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-istio-token" pod "8c720270-bcf8-433a-8edc-bb68f329640c" (UID: "8c720270-bcf8-433a-8edc-bb68f329640c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.788298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.788017    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:29:14.790020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.789964    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:29:14.824802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.824770    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-certs\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.825416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.825394    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-envoy\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.826042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.825266    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-certs" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.825987    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-token\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-token\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831355    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-kube-api-access-xwsbj\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-kube-api-access-xwsbj\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831401    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-token\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-token\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831482    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-podinfo\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831536    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istiod-ca-cert\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831618    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-socket\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831682    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-data\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831746    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-kube-api-access-bxfsw\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-kube-api-access-bxfsw\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831782    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-certs\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831838    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-data\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831865    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-credential-socket\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831926    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603e680b-f3b7-425b-83ac-b7e7051cd801-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603e680b-f3b7-425b-83ac-b7e7051cd801-istiod-ca-cert\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.831995    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-credential-socket\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.832034    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-envoy\") pod \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\" (UID: \"90b6695b-e9f4-4fd0-8979-d06d71d6a085\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.832124    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-socket\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.832160    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-podinfo\") pod \"603e680b-f3b7-425b-83ac-b7e7051cd801\" (UID: \"603e680b-f3b7-425b-83ac-b7e7051cd801\") "
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.832429    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istiod-ca-cert" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.833157    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c720270-bcf8-433a-8edc-bb68f329640c-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.833181    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.833196    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.833210    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5kx\" (UniqueName: \"kubernetes.io/projected/8c720270-bcf8-433a-8edc-bb68f329640c-kube-api-access-wr5kx\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.833260    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.834651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.833275    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c720270-bcf8-433a-8edc-bb68f329640c-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.837364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.836835    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-data" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.837364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.836907    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603e680b-f3b7-425b-83ac-b7e7051cd801-istiod-ca-cert" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.837364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.837031    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-certs" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.837364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.837071    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-socket" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.837364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.837172    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-credential-socket" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.837709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.837684    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-socket" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.837908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.837866    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-data" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.838323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.838298    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-credential-socket" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.853251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.852791    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-token" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.853403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.853301    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-kube-api-access-bxfsw" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "kube-api-access-bxfsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.853462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.853425    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-kube-api-access-xwsbj" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "kube-api-access-xwsbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.855275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.854091    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-envoy" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.855275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.854597    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-podinfo" pod "603e680b-f3b7-425b-83ac-b7e7051cd801" (UID: "603e680b-f3b7-425b-83ac-b7e7051cd801"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.855275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.855077    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-token" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.856709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.855966    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-envoy" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.856709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.856614    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-podinfo" pod "90b6695b-e9f4-4fd0-8979-d06d71d6a085" (UID: "90b6695b-e9f4-4fd0-8979-d06d71d6a085"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.878243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.877857    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.934765    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-podinfo\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.934831    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-credential-socket\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.934981    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-certs\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.935017    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-socket\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.935044    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-envoy\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.935082    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-data\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.935188    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istiod-ca-cert\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.935301    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-token\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-token\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.935336    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-kube-api-access-rl9rj\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-kube-api-access-rl9rj\") pod \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\" (UID: \"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf\") "
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.936086    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-socket" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.938961    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.938991    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939006    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwsbj\" (UniqueName: \"kubernetes.io/projected/603e680b-f3b7-425b-83ac-b7e7051cd801-kube-api-access-xwsbj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939020    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939033    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939045    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939058    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939073    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939087    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxfsw\" (UniqueName: \"kubernetes.io/projected/90b6695b-e9f4-4fd0-8979-d06d71d6a085-kube-api-access-bxfsw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939099    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939111    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939125    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939136    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603e680b-f3b7-425b-83ac-b7e7051cd801-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939148    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939159    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b6695b-e9f4-4fd0-8979-d06d71d6a085-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939172    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603e680b-f3b7-425b-83ac-b7e7051cd801-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.939183    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603e680b-f3b7-425b-83ac-b7e7051cd801-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.940503    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-credential-socket" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.940890    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istiod-ca-cert" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.941588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.941141    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-data" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.963703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.942086    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-podinfo" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.963703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.942408    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-certs" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.964044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.963944    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-envoy" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.964044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.963982    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-token" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.964044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:14.964022    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-kube-api-access-rl9rj" pod "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" (UID: "fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf"). InnerVolumeSpecName "kube-api-access-rl9rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:15.002818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.002784    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:29:15.013276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.013208    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"]
Oct 30 22:29:15.023531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.023431    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-97d66"]
Oct 30 22:29:15.039934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.039855    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-kube-api-access-hwcdb\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-kube-api-access-hwcdb\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.039934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.039894    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-credential-socket\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.040127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040041    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741a2f67-6328-40cb-8866-cc4e17b0584d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a2f67-6328-40cb-8866-cc4e17b0584d-istiod-ca-cert\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.040175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040146    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-certs\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.040263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040209    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-podinfo\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.040335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040234    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-credential-socket" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.040484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040461    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-certs" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.040545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040476    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741a2f67-6328-40cb-8866-cc4e17b0584d-istiod-ca-cert" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:15.040545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040487    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-data\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.040545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040532    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-token\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-token\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.040706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040563    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-envoy\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.040706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040637    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-socket\") pod \"741a2f67-6328-40cb-8866-cc4e17b0584d\" (UID: \"741a2f67-6328-40cb-8866-cc4e17b0584d\") "
Oct 30 22:29:15.040806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.040755    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-data" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.041058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041032    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-socket" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.041653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041631    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.041757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041661    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.041757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041679    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.041757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041695    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.041757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041710    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.041757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041724    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.041757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041738    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.041757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041750    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.042119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041762    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a2f67-6328-40cb-8866-cc4e17b0584d-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.042119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041777    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.042119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041790    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.042119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041802    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.042119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.041814    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl9rj\" (UniqueName: \"kubernetes.io/projected/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf-kube-api-access-rl9rj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.042947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.042920    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-envoy" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:15.043024 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.042950    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-kube-api-access-hwcdb" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "kube-api-access-hwcdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:15.043068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.043030    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-podinfo" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:15.043103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.043084    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-token" pod "741a2f67-6328-40cb-8866-cc4e17b0584d" (UID: "741a2f67-6328-40cb-8866-cc4e17b0584d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:15.143965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.143929    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.143965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.143963    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.144191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.143980    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwcdb\" (UniqueName: \"kubernetes.io/projected/741a2f67-6328-40cb-8866-cc4e17b0584d-kube-api-access-hwcdb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.144191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.143997    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a2f67-6328-40cb-8866-cc4e17b0584d-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:15.668898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.668863    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"
Oct 30 22:29:15.669723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.669651    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t" event={"ID":"90b6695b-e9f4-4fd0-8979-d06d71d6a085","Type":"ContainerDied","Data":"4e1df69a5509ce8a13178bfbfa7555809a6b1206bdd75f4af2141a2033ebba86"}
Oct 30 22:29:15.669858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.669727    4581 scope.go:122] "RemoveContainer" containerID="8acce8cd0e073153aa418904524a83822fcc64acf9947645e2051ad66000ea95"
Oct 30 22:29:15.671842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.671773    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj" event={"ID":"741a2f67-6328-40cb-8866-cc4e17b0584d","Type":"ContainerDied","Data":"d93f69b84025c054b31467859c340a13f3b5d4a3f67600a0b5a9b4346cee3d64"}
Oct 30 22:29:15.671925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.671899    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"
Oct 30 22:29:15.686379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.686325    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4" event={"ID":"603e680b-f3b7-425b-83ac-b7e7051cd801","Type":"ContainerDied","Data":"32f90b1398e4dc174d51e1391b38e0c8269932a71f58bf71ace38b9dfcd6ae25"}
Oct 30 22:29:15.690522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.689047    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"
Oct 30 22:29:15.693029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.690688    4581 scope.go:122] "RemoveContainer" containerID="7c71a65413e7ba1114212a26e7f33c30c8f8bc262be2b9eae706fb828ec29afe"
Oct 30 22:29:15.710120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.710078    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf" event={"ID":"fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf","Type":"ContainerDied","Data":"cf91f224173cf5f62fefc5d16ec19f1315bbab7ff292527f116b72d7ce4cc94c"}
Oct 30 22:29:15.710414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.710389    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"
Oct 30 22:29:15.736284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.735437    4581 scope.go:122] "RemoveContainer" containerID="762af472bc458080588b7192c41771ae4a4619665e2b6a0db4fa47e764bbcc13"
Oct 30 22:29:15.743534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.743498    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"]
Oct 30 22:29:15.766655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.763741    4581 scope.go:122] "RemoveContainer" containerID="99b0234912798ff5d5e761f2b8192181ed27f73519263d4f4f4ef6bbaf65ecdb"
Oct 30 22:29:15.766655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.764487    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-57vcj"]
Oct 30 22:29:15.793587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.788819    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"]
Oct 30 22:29:15.794210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.794182    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-gwz9t"]
Oct 30 22:29:15.861558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.861498    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"]
Oct 30 22:29:15.929507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.929368    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-97nr4"]
Oct 30 22:29:15.938965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.938900    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"]
Oct 30 22:29:15.947513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:15.947465    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hstmf"]
Oct 30 22:29:16.113914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:16.113880    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" path="/var/lib/kubelet/pods/603e680b-f3b7-425b-83ac-b7e7051cd801/volumes"
Oct 30 22:29:16.114621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:16.114598    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" path="/var/lib/kubelet/pods/741a2f67-6328-40cb-8866-cc4e17b0584d/volumes"
Oct 30 22:29:16.115291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:16.115268    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" path="/var/lib/kubelet/pods/8c720270-bcf8-433a-8edc-bb68f329640c/volumes"
Oct 30 22:29:16.116339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:16.116322    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" path="/var/lib/kubelet/pods/90b6695b-e9f4-4fd0-8979-d06d71d6a085/volumes"
Oct 30 22:29:16.116952 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:16.116933    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" path="/var/lib/kubelet/pods/fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf/volumes"
Oct 30 22:29:19.013831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:19.013774    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": read tcp 10.133.0.2:37448->10.133.0.160:15021: read: connection reset by peer" start-of-body=
Oct 30 22:29:19.014383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:19.013877    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": read tcp 10.133.0.2:37448->10.133.0.160:15021: read: connection reset by peer"
Oct 30 22:29:19.772742 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:19.772699    4581 generic.go:356] "Generic (PLEG): container finished" podID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerID="ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be" exitCode=0
Oct 30 22:29:19.772955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:19.772777    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" event={"ID":"9c0e492e-4281-4be3-bed8-8f57b2f13eb5","Type":"ContainerDied","Data":"ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be"}
Oct 30 22:29:19.773359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:19.773338    4581 scope.go:122] "RemoveContainer" containerID="ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be"
Oct 30 22:29:19.775211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:19.775170    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:29:19.775344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:19.775291    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:29:19.775344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:19.775335    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:29:20.797268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:20.797212    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" event={"ID":"9c0e492e-4281-4be3-bed8-8f57b2f13eb5","Type":"ContainerStarted","Data":"93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45"}
Oct 30 22:29:21.338365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:21.338329    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:29:21.338616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:21.338591    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:29:21.338712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:21.338645    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:29:22.338673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.338637    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:29:22.339102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.338690    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:29:22.904713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.904680    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"]
Oct 30 22:29:22.905555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905534    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4f5ba764-e1df-4205-aab3-2a79e9168fe6" containerName="revoke-keys"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905557    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5ba764-e1df-4205-aab3-2a79e9168fe6" containerName="revoke-keys"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905573    4581 cpu_manager.go:414] "containerMap: removing container" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905578    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905599    4581 cpu_manager.go:414] "containerMap: removing container" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905605    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905616    4581 cpu_manager.go:414] "containerMap: removing container" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905622    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905645    4581 cpu_manager.go:414] "containerMap: removing container" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905650    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905656    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy"
Oct 30 22:29:22.905666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905661    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy"
Oct 30 22:29:22.905990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905777    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b6695b-e9f4-4fd0-8979-d06d71d6a085" containerName="istio-proxy"
Oct 30 22:29:22.905990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905787    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c720270-bcf8-433a-8edc-bb68f329640c" containerName="istio-proxy"
Oct 30 22:29:22.905990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905794    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5ba764-e1df-4205-aab3-2a79e9168fe6" containerName="revoke-keys"
Oct 30 22:29:22.905990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905801    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="603e680b-f3b7-425b-83ac-b7e7051cd801" containerName="istio-proxy"
Oct 30 22:29:22.905990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905809    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="741a2f67-6328-40cb-8866-cc4e17b0584d" containerName="istio-proxy"
Oct 30 22:29:22.905990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.905816    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9b24a3-c79b-4d90-80b3-c3c6be2e94cf" containerName="istio-proxy"
Oct 30 22:29:22.910615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.910593    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:22.923784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.923752    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f443d6-openshift-default-dockercfg-lkjnj\""
Oct 30 22:29:22.946073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:22.946042    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"]
Oct 30 22:29:23.043555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.043519    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7npc\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-kube-api-access-d7npc\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.043797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.043570    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-token\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.043797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.043700    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.043797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.043747    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-envoy\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.043797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.043772    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-data\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.043951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.043896    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-credential-socket\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.044023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.044009    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c8d384-92f1-40f3-a233-84519c2ee72f-istiod-ca-cert\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.044094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.044079    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-socket\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.044130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.044114    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-certs\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.158511    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7npc\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-kube-api-access-d7npc\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.158594    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-token\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.158702    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.158742    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-envoy\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.158769    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-data\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.158855    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-credential-socket\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.158929    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c8d384-92f1-40f3-a233-84519c2ee72f-istiod-ca-cert\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.159030    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-socket\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.159368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.159061    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-certs\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.171237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.166732    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c8d384-92f1-40f3-a233-84519c2ee72f-istiod-ca-cert\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.171237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.167349    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-certs\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.171237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.167577    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-socket\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.171237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.167797    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-credential-socket\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.171237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.168015    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-data\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.223273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.223238    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7npc\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-kube-api-access-d7npc\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.223741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.223722    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-envoy\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.240941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.240852    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.241667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.241641    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-token\") pod \"e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") " pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.348244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.338616    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:29:23.348244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.338678    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:29:23.533432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.533303    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:23.932662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.932634    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"]
Oct 30 22:29:23.933939 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:29:23.933912    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c8d384_92f1_40f3_a233_84519c2ee72f.slice/crio-cce8a111cf098e3bab9448b15af1e2ffc261fe792e1130c3a752d3979f811083 WatchSource:0}: Error finding container cce8a111cf098e3bab9448b15af1e2ffc261fe792e1130c3a752d3979f811083: Status 404 returned error can't find the container with id cce8a111cf098e3bab9448b15af1e2ffc261fe792e1130c3a752d3979f811083
Oct 30 22:29:23.936799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.936762    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:29:23.936887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.936840    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:29:23.936887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:23.936873    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:29:24.347291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:24.342638    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:29:24.347291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:24.342714    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:29:24.848448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:24.848416    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" event={"ID":"c3c8d384-92f1-40f3-a233-84519c2ee72f","Type":"ContainerStarted","Data":"6051edd0e2f2dac4628d09b571d39054fb76b2ad37c7e8e90abd0082cd5ed4da"}
Oct 30 22:29:24.848914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:24.848893    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" event={"ID":"c3c8d384-92f1-40f3-a233-84519c2ee72f","Type":"ContainerStarted","Data":"cce8a111cf098e3bab9448b15af1e2ffc261fe792e1130c3a752d3979f811083"}
Oct 30 22:29:24.917309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:24.914888    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podStartSLOduration=2.914870804 podStartE2EDuration="2.914870804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:24.904390346 +0000 UTC m=+1348.943329233" watchObservedRunningTime="2026-10-30 22:29:24.914870804 +0000 UTC m=+1348.953809689"
Oct 30 22:29:25.338549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:25.338514    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:29:25.338751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:25.338576    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:29:25.533929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:25.533897    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:25.540104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:25.540072    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:25.885852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:25.885816    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:25.887329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:25.887301    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:29:26.338260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:26.338163    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:29:26.338414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:26.338258    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:29:27.342950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:27.342921    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:29:27.343480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:27.343465    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:29:27.344195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:27.344172    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 22:29:29.214493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.213438    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"]
Oct 30 22:29:29.223298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.223275    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:29.228120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.228088    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"]
Oct 30 22:29:29.236483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.236458    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f443d6\""
Oct 30 22:29:29.251358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.251327    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"]
Oct 30 22:29:29.257037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.257011    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:29.270489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.270463    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f443d6-dockercfg-n5r4r\""
Oct 30 22:29:29.287337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.287306    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"]
Oct 30 22:29:29.312422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.312384    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm\" (UID: \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\") " pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:29.313165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.312856    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfg54\" (UniqueName: \"kubernetes.io/projected/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-kube-api-access-vfg54\") pod \"payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm\" (UID: \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\") " pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:29.430243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.427297    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5\" (UID: \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:29.430243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.427485    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxhs\" (UniqueName: \"kubernetes.io/projected/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-kube-api-access-qsxhs\") pod \"payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5\" (UID: \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:29.430243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.427584    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm\" (UID: \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\") " pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:29.430243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.427669    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfg54\" (UniqueName: \"kubernetes.io/projected/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-kube-api-access-vfg54\") pod \"payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm\" (UID: \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\") " pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:29.430243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.428876    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm\" (UID: \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\") " pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:29.463240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.457947    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfg54\" (UniqueName: \"kubernetes.io/projected/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-kube-api-access-vfg54\") pod \"payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm\" (UID: \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\") " pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:29.536614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.536511    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5\" (UID: \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:29.537014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.536992    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxhs\" (UniqueName: \"kubernetes.io/projected/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-kube-api-access-qsxhs\") pod \"payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5\" (UID: \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:29.538904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.538877    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5\" (UID: \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:29.546009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.545980    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:29.670030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.666582    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxhs\" (UniqueName: \"kubernetes.io/projected/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-kube-api-access-qsxhs\") pod \"payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5\" (UID: \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:29.876788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.876752    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"]
Oct 30 22:29:29.881657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.881628    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-wfsw5"]
Oct 30 22:29:29.887759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:29.887730    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:30.042860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.042809    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"]
Oct 30 22:29:30.044911 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:29:30.044873    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb7fc7d_8e84_4012_8ed1_28b1516d99ad.slice/crio-1346485ba9d9f7df71dc2b4b93b9d0265a6df091427e2703e4560598f956c811 WatchSource:0}: Error finding container 1346485ba9d9f7df71dc2b4b93b9d0265a6df091427e2703e4560598f956c811: Status 404 returned error can't find the container with id 1346485ba9d9f7df71dc2b4b93b9d0265a6df091427e2703e4560598f956c811
Oct 30 22:29:30.100425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.100387    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3fbf76-da83-476a-89c2-3cac5abee640" path="/var/lib/kubelet/pods/1a3fbf76-da83-476a-89c2-3cac5abee640/volumes"
Oct 30 22:29:30.225845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.225819    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"]
Oct 30 22:29:30.228186 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:29:30.228159    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b2db7c_eaf7_4bcb_9075_e6ff0c2f9e14.slice/crio-8b55cc52df10178b202b2d9fe241dfb2be24b66ffe69bae00c76f5d036b0e006 WatchSource:0}: Error finding container 8b55cc52df10178b202b2d9fe241dfb2be24b66ffe69bae00c76f5d036b0e006: Status 404 returned error can't find the container with id 8b55cc52df10178b202b2d9fe241dfb2be24b66ffe69bae00c76f5d036b0e006
Oct 30 22:29:30.965000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.964935    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5" event={"ID":"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14","Type":"ContainerStarted","Data":"01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e"}
Oct 30 22:29:30.965211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.965020    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5" event={"ID":"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14","Type":"ContainerStarted","Data":"8b55cc52df10178b202b2d9fe241dfb2be24b66ffe69bae00c76f5d036b0e006"}
Oct 30 22:29:30.965211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.965051    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:30.966870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.966843    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm" event={"ID":"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad","Type":"ContainerStarted","Data":"c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b"}
Oct 30 22:29:30.966993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.966896    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm" event={"ID":"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad","Type":"ContainerStarted","Data":"1346485ba9d9f7df71dc2b4b93b9d0265a6df091427e2703e4560598f956c811"}
Oct 30 22:29:30.966993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:30.966948    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:31.004066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:31.003999    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5" podStartSLOduration=2.003977597 podStartE2EDuration="2.003977597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:30.998739593 +0000 UTC m=+1355.037678480" watchObservedRunningTime="2026-10-30 22:29:31.003977597 +0000 UTC m=+1355.042916484"
Oct 30 22:29:31.032707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:31.032651    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm" podStartSLOduration=2.032630974 podStartE2EDuration="2.032630974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:31.031549089 +0000 UTC m=+1355.070487976" watchObservedRunningTime="2026-10-30 22:29:31.032630974 +0000 UTC m=+1355.071569861"
Oct 30 22:29:38.276240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.275815    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"]
Oct 30 22:29:38.282270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.282239    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:38.304370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.302477    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"]
Oct 30 22:29:38.369238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.368673    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2949c143\""
Oct 30 22:29:38.369238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.368836    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2949c143-dockercfg-mmm8r\""
Oct 30 22:29:38.611364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.573335    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmkrm\" (UniqueName: \"kubernetes.io/projected/af752c40-842e-4c32-990f-da63c2a0066f-kube-api-access-xmkrm\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8\" (UID: \"af752c40-842e-4c32-990f-da63c2a0066f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:38.611364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.573674    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af752c40-842e-4c32-990f-da63c2a0066f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8\" (UID: \"af752c40-842e-4c32-990f-da63c2a0066f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:38.676655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.676622    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af752c40-842e-4c32-990f-da63c2a0066f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8\" (UID: \"af752c40-842e-4c32-990f-da63c2a0066f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:38.676875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.676713    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkrm\" (UniqueName: \"kubernetes.io/projected/af752c40-842e-4c32-990f-da63c2a0066f-kube-api-access-xmkrm\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8\" (UID: \"af752c40-842e-4c32-990f-da63c2a0066f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:38.677896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.677867    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af752c40-842e-4c32-990f-da63c2a0066f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8\" (UID: \"af752c40-842e-4c32-990f-da63c2a0066f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:38.695046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.695013    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkrm\" (UniqueName: \"kubernetes.io/projected/af752c40-842e-4c32-990f-da63c2a0066f-kube-api-access-xmkrm\") pod \"payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8\" (UID: \"af752c40-842e-4c32-990f-da63c2a0066f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:38.708835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:38.708785    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:39.038670 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:39.038638    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"]
Oct 30 22:29:39.039774 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:29:39.039742    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf752c40_842e_4c32_990f_da63c2a0066f.slice/crio-3849b90a4c0032838863f95aceceb012e9c6c0cbd13742ed804630b41a92669f WatchSource:0}: Error finding container 3849b90a4c0032838863f95aceceb012e9c6c0cbd13742ed804630b41a92669f: Status 404 returned error can't find the container with id 3849b90a4c0032838863f95aceceb012e9c6c0cbd13742ed804630b41a92669f
Oct 30 22:29:39.616587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:39.616548    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"]
Oct 30 22:29:39.617052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:39.616754    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy" containerID="cri-o://bc66dad1c0a2c6c301e4b6d7d1814c7202ca7e66a9e62e912948bc909c155e89" gracePeriod=30
Oct 30 22:29:39.625652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:39.625612    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"]
Oct 30 22:29:39.626835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:39.625936    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy" containerID="cri-o://d6b2f60034a05403bff29c2182833d2b3f50aa45df9f0aa9004be58d4c9735a4" gracePeriod=30
Oct 30 22:29:40.060686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:40.060583    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8" event={"ID":"af752c40-842e-4c32-990f-da63c2a0066f","Type":"ContainerStarted","Data":"dd708a2058bf4dca6b0a7d7802179ceb96ee349064f1b0b67f975b518d036e54"}
Oct 30 22:29:40.060686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:40.060644    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8" event={"ID":"af752c40-842e-4c32-990f-da63c2a0066f","Type":"ContainerStarted","Data":"3849b90a4c0032838863f95aceceb012e9c6c0cbd13742ed804630b41a92669f"}
Oct 30 22:29:40.060935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:40.060814    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:40.100231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:40.100157    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8" podStartSLOduration=2.10013847 podStartE2EDuration="2.10013847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:40.095556078 +0000 UTC m=+1364.134494969" watchObservedRunningTime="2026-10-30 22:29:40.10013847 +0000 UTC m=+1364.139077358"
Oct 30 22:29:40.822571 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:29:40.822533    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:29:41.980467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:41.980437    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:29:41.980979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:41.980516    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:29:43.919166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.919130    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"]
Oct 30 22:29:43.923930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.923906    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:43.937705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.937655    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:43.943997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.943969    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"]
Oct 30 22:29:43.947263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.947240    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:43.947478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.947461    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:43.947587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.947513    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:29:43.994466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.994414    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:43.994675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.994520    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:43.994761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:43.994702    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwnn\" (UniqueName: \"kubernetes.io/projected/031a4fef-a653-4c37-955f-d1d7aec77833-kube-api-access-hnwnn\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:44.096007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:44.095978    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:44.096259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:44.096039    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:44.096259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:44.096153    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwnn\" (UniqueName: \"kubernetes.io/projected/031a4fef-a653-4c37-955f-d1d7aec77833-kube-api-access-hnwnn\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:44.096984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:44.096957    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:44.099338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:44.099313    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:44.113548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:44.113520    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwnn\" (UniqueName: \"kubernetes.io/projected/031a4fef-a653-4c37-955f-d1d7aec77833-kube-api-access-hnwnn\") pod \"maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:44.240089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:44.240004    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:44.566710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:44.566612    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"]
Oct 30 22:29:44.570493 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:29:44.570437    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031a4fef_a653_4c37_955f_d1d7aec77833.slice/crio-6e4219a6f3538b1868f7dfd03a80ec8abd5451c36b88181a24e1b6e61c1813c6 WatchSource:0}: Error finding container 6e4219a6f3538b1868f7dfd03a80ec8abd5451c36b88181a24e1b6e61c1813c6: Status 404 returned error can't find the container with id 6e4219a6f3538b1868f7dfd03a80ec8abd5451c36b88181a24e1b6e61c1813c6
Oct 30 22:29:45.636672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.636598    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" event={"ID":"4485c1fc-ee74-4a23-a985-d74366141b6e","Type":"ContainerDied","Data":"bc66dad1c0a2c6c301e4b6d7d1814c7202ca7e66a9e62e912948bc909c155e89"}
Oct 30 22:29:45.667443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.665430    4581 generic.go:356] "Generic (PLEG): container finished" podID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerID="bc66dad1c0a2c6c301e4b6d7d1814c7202ca7e66a9e62e912948bc909c155e89" exitCode=0
Oct 30 22:29:45.671129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.670666    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.837584    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-credential-socket\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.837742    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-kube-api-access-j2f5v\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-kube-api-access-j2f5v\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.837799    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-envoy\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.837945    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-certs\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.838037    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-data\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.838141    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-token\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-token\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.838186    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-socket\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.838237    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-podinfo\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.838297    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istiod-ca-cert\") pod \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\" (UID: \"d5dda98e-6ca9-4944-8d0d-1f9a9380835a\") "
Oct 30 22:29:45.858238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.856601    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-credential-socket" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.873028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.860805    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istiod-ca-cert" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:45.873028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.861635    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt" event={"ID":"031a4fef-a653-4c37-955f-d1d7aec77833","Type":"ContainerStarted","Data":"6e4219a6f3538b1868f7dfd03a80ec8abd5451c36b88181a24e1b6e61c1813c6"}
Oct 30 22:29:45.887302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.881505    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-certs" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.887302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.881662    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-kube-api-access-j2f5v" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "kube-api-access-j2f5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:45.894099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.889296    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-data" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.894099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.893303    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-socket" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.896192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.896151    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-envoy" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.951268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.940939    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:45.951268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.940976    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:45.951268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.940991    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:45.951268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.941005    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:45.951268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.941022    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:45.951268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.941040    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2f5v\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-kube-api-access-j2f5v\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:45.951268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.941057    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.027246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:45.976618    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:29:46.070268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.066865    4581 generic.go:356] "Generic (PLEG): container finished" podID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerID="d6b2f60034a05403bff29c2182833d2b3f50aa45df9f0aa9004be58d4c9735a4" exitCode=0
Oct 30 22:29:46.070268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.066915    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc" event={"ID":"d5dda98e-6ca9-4944-8d0d-1f9a9380835a","Type":"ContainerDied","Data":"d6b2f60034a05403bff29c2182833d2b3f50aa45df9f0aa9004be58d4c9735a4"}
Oct 30 22:29:46.070268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.066973    4581 scope.go:122] "RemoveContainer" containerID="d6b2f60034a05403bff29c2182833d2b3f50aa45df9f0aa9004be58d4c9735a4"
Oct 30 22:29:46.070268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.067185    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"
Oct 30 22:29:46.247243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.243598    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-kube-api-access-xvvbc\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-kube-api-access-xvvbc\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.247243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.243757    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-credential-socket\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.247243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.243796    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-socket\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.279266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.247457    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-envoy\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.279462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.279290    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-podinfo\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.279462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.279407    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-token\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-token\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.279596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.279492    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-data\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.283246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.279803    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-certs\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.283246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.279875    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4485c1fc-ee74-4a23-a985-d74366141b6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4485c1fc-ee74-4a23-a985-d74366141b6e-istiod-ca-cert\") pod \"4485c1fc-ee74-4a23-a985-d74366141b6e\" (UID: \"4485c1fc-ee74-4a23-a985-d74366141b6e\") "
Oct 30 22:29:46.305338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.301659    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-socket" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:46.311056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.310881    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-data" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:46.313268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.311259    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-certs" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:46.329911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.329873    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4485c1fc-ee74-4a23-a985-d74366141b6e-istiod-ca-cert" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:46.392027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.391274    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-envoy" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:46.403031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.394346    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-podinfo" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:46.405510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.405479    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-kube-api-access-xvvbc" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "kube-api-access-xvvbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:46.430593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.416610    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-credential-socket" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:46.430795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.426489    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-token" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:46.430935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.428673    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.431077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.431060    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.431254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.431240    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4485c1fc-ee74-4a23-a985-d74366141b6e-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.431437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.431382    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.431556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.431542    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.431679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.431667    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.445518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.445478    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-podinfo" pod "4485c1fc-ee74-4a23-a985-d74366141b6e" (UID: "4485c1fc-ee74-4a23-a985-d74366141b6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:46.527454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.527389    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-token" pod "d5dda98e-6ca9-4944-8d0d-1f9a9380835a" (UID: "d5dda98e-6ca9-4944-8d0d-1f9a9380835a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:46.543251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.536069    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvbc\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-kube-api-access-xvvbc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.543251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.536107    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4485c1fc-ee74-4a23-a985-d74366141b6e-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.543251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.536124    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.543251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.536138    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4485c1fc-ee74-4a23-a985-d74366141b6e-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:46.543251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:46.536151    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dda98e-6ca9-4944-8d0d-1f9a9380835a-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:47.217243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.216291    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"]
Oct 30 22:29:47.289030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.288974    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-mb6lc"]
Oct 30 22:29:47.330826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.330721    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2" event={"ID":"4485c1fc-ee74-4a23-a985-d74366141b6e","Type":"ContainerDied","Data":"8272f83056b1c0d36a9a4284ccc201c2256cc824d8f3d06c522b31b4244ce658"}
Oct 30 22:29:47.331104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.331084    4581 scope.go:122] "RemoveContainer" containerID="bc66dad1c0a2c6c301e4b6d7d1814c7202ca7e66a9e62e912948bc909c155e89"
Oct 30 22:29:47.331529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.331495    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"
Oct 30 22:29:47.385509 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.385459    4581 generic.go:356] "Generic (PLEG): container finished" podID="031a4fef-a653-4c37-955f-d1d7aec77833" containerID="eb9bed9b747d26326b23f7cb9e8a59f167671e8dd817fc024ff0b36cf012d1b2" exitCode=0
Oct 30 22:29:47.385886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.385863    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt" event={"ID":"031a4fef-a653-4c37-955f-d1d7aec77833","Type":"ContainerDied","Data":"eb9bed9b747d26326b23f7cb9e8a59f167671e8dd817fc024ff0b36cf012d1b2"}
Oct 30 22:29:47.482334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.482293    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"]
Oct 30 22:29:47.508355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.508312    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:29:47.559252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.557632    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-6d9p2"]
Oct 30 22:29:47.659814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.659771    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:29:47.806238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.801404    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:29:47.847250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.846357    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:29:47.847250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:47.846444    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:29:48.144636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:48.144600    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" path="/var/lib/kubelet/pods/4485c1fc-ee74-4a23-a985-d74366141b6e/volumes"
Oct 30 22:29:48.145208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:48.145191    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" path="/var/lib/kubelet/pods/d5dda98e-6ca9-4944-8d0d-1f9a9380835a/volumes"
Oct 30 22:29:48.982255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:48.972414    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:49.027240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.026385    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-xks-ca\") pod \"031a4fef-a653-4c37-955f-d1d7aec77833\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") "
Oct 30 22:29:49.027240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.026472    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031a4fef-a653-4c37-955f-d1d7aec77833-kube-api-access-hnwnn\" (UniqueName: \"kubernetes.io/projected/031a4fef-a653-4c37-955f-d1d7aec77833-kube-api-access-hnwnn\") pod \"031a4fef-a653-4c37-955f-d1d7aec77833\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") "
Oct 30 22:29:49.027240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.026528    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-openshift-ca\") pod \"031a4fef-a653-4c37-955f-d1d7aec77833\" (UID: \"031a4fef-a653-4c37-955f-d1d7aec77833\") "
Oct 30 22:29:49.028455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.027874    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-openshift-ca" pod "031a4fef-a653-4c37-955f-d1d7aec77833" (UID: "031a4fef-a653-4c37-955f-d1d7aec77833"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:49.032806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.032776    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031a4fef-a653-4c37-955f-d1d7aec77833-kube-api-access-hnwnn" pod "031a4fef-a653-4c37-955f-d1d7aec77833" (UID: "031a4fef-a653-4c37-955f-d1d7aec77833"). InnerVolumeSpecName "kube-api-access-hnwnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:49.036003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.035445    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-xks-ca" pod "031a4fef-a653-4c37-955f-d1d7aec77833" (UID: "031a4fef-a653-4c37-955f-d1d7aec77833"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:49.132466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.132380    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:49.132740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.132719    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwnn\" (UniqueName: \"kubernetes.io/projected/031a4fef-a653-4c37-955f-d1d7aec77833-kube-api-access-hnwnn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:49.132870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.132856    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031a4fef-a653-4c37-955f-d1d7aec77833-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:49.551834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.551742    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"
Oct 30 22:29:49.552025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.551999    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt" event={"ID":"031a4fef-a653-4c37-955f-d1d7aec77833","Type":"ContainerDied","Data":"6e4219a6f3538b1868f7dfd03a80ec8abd5451c36b88181a24e1b6e61c1813c6"}
Oct 30 22:29:49.552141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:49.552129    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e4219a6f3538b1868f7dfd03a80ec8abd5451c36b88181a24e1b6e61c1813c6"
Oct 30 22:29:51.075915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:51.075887    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:29:54.378230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:54.378189    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"]
Oct 30 22:29:55.652617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.652577    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"]
Oct 30 22:29:55.653558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653535    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy"
Oct 30 22:29:55.653558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653558    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy"
Oct 30 22:29:55.653734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653587    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy"
Oct 30 22:29:55.653734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653596    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy"
Oct 30 22:29:55.653734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653615    4581 cpu_manager.go:414] "containerMap: removing container" podUID="031a4fef-a653-4c37-955f-d1d7aec77833" containerName="revoke-keys"
Oct 30 22:29:55.653734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653622    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031a4fef-a653-4c37-955f-d1d7aec77833" containerName="revoke-keys"
Oct 30 22:29:55.653931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653811    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4485c1fc-ee74-4a23-a985-d74366141b6e" containerName="istio-proxy"
Oct 30 22:29:55.653931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653830    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="031a4fef-a653-4c37-955f-d1d7aec77833" containerName="revoke-keys"
Oct 30 22:29:55.653931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.653852    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dda98e-6ca9-4944-8d0d-1f9a9380835a" containerName="istio-proxy"
Oct 30 22:29:55.658100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.658078    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:55.674112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.674081    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-de31dd\""
Oct 30 22:29:55.680744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.680709    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"]
Oct 30 22:29:55.789921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.789863    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9efcdb-4dbf-4736-951b-66a2baece0e1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z\" (UID: \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:55.790418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.790381    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cqr7\" (UniqueName: \"kubernetes.io/projected/3c9efcdb-4dbf-4736-951b-66a2baece0e1-kube-api-access-9cqr7\") pod \"payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z\" (UID: \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:55.907476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.906409    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9efcdb-4dbf-4736-951b-66a2baece0e1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z\" (UID: \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:55.907476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.906612    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cqr7\" (UniqueName: \"kubernetes.io/projected/3c9efcdb-4dbf-4736-951b-66a2baece0e1-kube-api-access-9cqr7\") pod \"payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z\" (UID: \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:55.930616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:55.907976    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9efcdb-4dbf-4736-951b-66a2baece0e1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z\" (UID: \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:56.034072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:56.034018    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cqr7\" (UniqueName: \"kubernetes.io/projected/3c9efcdb-4dbf-4736-951b-66a2baece0e1-kube-api-access-9cqr7\") pod \"payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z\" (UID: \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\") " pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:56.274446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:56.274330    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:56.589939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:56.589910    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"]
Oct 30 22:29:56.592577 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:29:56.592536    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9efcdb_4dbf_4736_951b_66a2baece0e1.slice/crio-209e77a495e978d0fbd0086406699efa41842fac3f9b30e453c9008d9459e244 WatchSource:0}: Error finding container 209e77a495e978d0fbd0086406699efa41842fac3f9b30e453c9008d9459e244: Status 404 returned error can't find the container with id 209e77a495e978d0fbd0086406699efa41842fac3f9b30e453c9008d9459e244
Oct 30 22:29:56.778873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:56.778839    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z" event={"ID":"3c9efcdb-4dbf-4736-951b-66a2baece0e1","Type":"ContainerStarted","Data":"209e77a495e978d0fbd0086406699efa41842fac3f9b30e453c9008d9459e244"}
Oct 30 22:29:57.804240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:57.799097    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z" event={"ID":"3c9efcdb-4dbf-4736-951b-66a2baece0e1","Type":"ContainerStarted","Data":"1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724"}
Oct 30 22:29:57.804240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:57.800365    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:29:57.837633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:57.837565    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z" podStartSLOduration=2.837550417 podStartE2EDuration="2.837550417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:57.833753249 +0000 UTC m=+1381.872692147" watchObservedRunningTime="2026-10-30 22:29:57.837550417 +0000 UTC m=+1381.876489303"
Oct 30 22:29:58.118333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:58.118294    4581 scope.go:122] "RemoveContainer" containerID="438956ccb3a63f989d5f2c8dc0314f1ded7e091e08fb29b525e65041fa1b964c"
Oct 30 22:29:58.130328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:58.130307    4581 scope.go:122] "RemoveContainer" containerID="248361ce54e3a089fc121b158586b4827f9ea0b786cb7cfe2b12573a6871a616"
Oct 30 22:29:58.150953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:58.150923    4581 scope.go:122] "RemoveContainer" containerID="60b8bc7f1fba14003e462f0eb5fba160ec1040da68d24439ae2b9fd843d3b058"
Oct 30 22:29:58.170177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:58.170148    4581 scope.go:122] "RemoveContainer" containerID="ad4e53ca929d7c789edbba49c32612b2b39e12196c4f86a0de0e8049b9904171"
Oct 30 22:29:58.186121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:58.186096    4581 scope.go:122] "RemoveContainer" containerID="255277d4f74115ed93e6178bdcfedab10ddb6cecdf947941e3b6bf89aec5b0f5"
Oct 30 22:29:58.198446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:58.198414    4581 scope.go:122] "RemoveContainer" containerID="c685ee6256b5aa6c554bee731c09fa8f9941f453e140badc1061af848f5542d3"
Oct 30 22:29:58.209366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:58.209342    4581 scope.go:122] "RemoveContainer" containerID="454955ff9d6f95841260dcd7f4749dc64c07c34ffd9ecda2ee3c95dde34491eb"
Oct 30 22:29:58.219687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:58.219666    4581 scope.go:122] "RemoveContainer" containerID="5b63d640a7b5d342192123a29d4601f42dca40cce17d780acec23676c9f194e8"
Oct 30 22:29:59.378632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.378585    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4" podUID="ec9ce949-0e67-4b71-8977-94cd796853cb" containerName="payload-pre-processing" containerID="cri-o://f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238" gracePeriod=25
Oct 30 22:29:59.706520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.706491    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:29:59.818363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.818330    4581 generic.go:356] "Generic (PLEG): container finished" podID="ec9ce949-0e67-4b71-8977-94cd796853cb" containerID="f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238" exitCode=0
Oct 30 22:29:59.818532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.818375    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"
Oct 30 22:29:59.818532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.818408    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4" event={"ID":"ec9ce949-0e67-4b71-8977-94cd796853cb","Type":"ContainerDied","Data":"f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238"}
Oct 30 22:29:59.818532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.818459    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4" event={"ID":"ec9ce949-0e67-4b71-8977-94cd796853cb","Type":"ContainerDied","Data":"e39ea75b80e21c1f947ec4ce89da64ce146e28d19a7533ce962f6eead57beba8"}
Oct 30 22:29:59.818532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.818481    4581 scope.go:122] "RemoveContainer" containerID="f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238"
Oct 30 22:29:59.830069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.830049    4581 scope.go:122] "RemoveContainer" containerID="f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238"
Oct 30 22:29:59.830407 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:29:59.830383    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238\": container with ID starting with f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238 not found: ID does not exist" containerID="f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238"
Oct 30 22:29:59.830538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.830416    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238"} err="failed to get container status \"f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238\": rpc error: code = NotFound desc = could not find container \"f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238\": container with ID starting with f3ec9cafd85b02ac4c5193b899d21ab7dd3024dfe87d82a3792cbd4e94704238 not found: ID does not exist"
Oct 30 22:29:59.830538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.830463    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9ce949-0e67-4b71-8977-94cd796853cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9ce949-0e67-4b71-8977-94cd796853cb-plugins-config-volume\") pod \"ec9ce949-0e67-4b71-8977-94cd796853cb\" (UID: \"ec9ce949-0e67-4b71-8977-94cd796853cb\") "
Oct 30 22:29:59.830722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.830703    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9ce949-0e67-4b71-8977-94cd796853cb-kube-api-access-phbvl\" (UniqueName: \"kubernetes.io/projected/ec9ce949-0e67-4b71-8977-94cd796853cb-kube-api-access-phbvl\") pod \"ec9ce949-0e67-4b71-8977-94cd796853cb\" (UID: \"ec9ce949-0e67-4b71-8977-94cd796853cb\") "
Oct 30 22:29:59.830870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.830847    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9ce949-0e67-4b71-8977-94cd796853cb-plugins-config-volume" pod "ec9ce949-0e67-4b71-8977-94cd796853cb" (UID: "ec9ce949-0e67-4b71-8977-94cd796853cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:59.831378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.831360    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9ce949-0e67-4b71-8977-94cd796853cb-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:29:59.833046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.833024    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9ce949-0e67-4b71-8977-94cd796853cb-kube-api-access-phbvl" pod "ec9ce949-0e67-4b71-8977-94cd796853cb" (UID: "ec9ce949-0e67-4b71-8977-94cd796853cb"). InnerVolumeSpecName "kube-api-access-phbvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:59.932964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:29:59.932876    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbvl\" (UniqueName: \"kubernetes.io/projected/ec9ce949-0e67-4b71-8977-94cd796853cb-kube-api-access-phbvl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:00.140876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.140839    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"]
Oct 30 22:30:00.141834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.141727    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ec9ce949-0e67-4b71-8977-94cd796853cb" containerName="payload-pre-processing"
Oct 30 22:30:00.141834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.141751    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9ce949-0e67-4b71-8977-94cd796853cb" containerName="payload-pre-processing"
Oct 30 22:30:00.142043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.141970    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9ce949-0e67-4b71-8977-94cd796853cb" containerName="payload-pre-processing"
Oct 30 22:30:00.161362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.161308    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"]
Oct 30 22:30:00.161566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.161395    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:00.185929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.181165    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:00.185929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.181337    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:00.185929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.181488    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:00.201719 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.194779    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:30:00.243456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.243412    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9bc499-e4f5-42ba-ad63-9a98580a7838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt\" (UID: \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:00.243650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.243495    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec931d7-85d3-48eb-9597-4e66cda196a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4\" (UID: \"aec931d7-85d3-48eb-9597-4e66cda196a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:00.243650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.243542    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78st\" (UniqueName: \"kubernetes.io/projected/2b9bc499-e4f5-42ba-ad63-9a98580a7838-kube-api-access-n78st\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt\" (UID: \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:00.243777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.243706    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbd28\" (UniqueName: \"kubernetes.io/projected/aec931d7-85d3-48eb-9597-4e66cda196a1-kube-api-access-bbd28\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4\" (UID: \"aec931d7-85d3-48eb-9597-4e66cda196a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:00.243876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.243857    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"]
Oct 30 22:30:00.259239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.256253    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"]
Oct 30 22:30:00.259239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.256352    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:00.274172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.274118    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"]
Oct 30 22:30:00.277942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.277914    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:00.287320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.286272    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:00.288326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.288300    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"]
Oct 30 22:30:00.297761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.297718    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"]
Oct 30 22:30:00.297761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.297754    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"]
Oct 30 22:30:00.297960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.297849    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:00.300690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.300666    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"]
Oct 30 22:30:00.306439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.306389    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:00.317002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.316873    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"]
Oct 30 22:30:00.323838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.323787    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"]
Oct 30 22:30:00.323948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.323893    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:00.329498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.329474    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d399fb07-755755d479-wsns4"]
Oct 30 22:30:00.329637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.329616    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:00.333929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.333906    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"]
Oct 30 22:30:00.339099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.339076    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"]
Oct 30 22:30:00.343978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.343954    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"]
Oct 30 22:30:00.344697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.344676    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5\" (UID: \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:00.344822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.344713    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9bc499-e4f5-42ba-ad63-9a98580a7838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt\" (UID: \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:00.344822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.344744    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cae3ab-37b3-4947-9e3c-b37bb36318e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l\" (UID: \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:00.344945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.344843    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec931d7-85d3-48eb-9597-4e66cda196a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4\" (UID: \"aec931d7-85d3-48eb-9597-4e66cda196a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:00.344945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.344893    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78st\" (UniqueName: \"kubernetes.io/projected/2b9bc499-e4f5-42ba-ad63-9a98580a7838-kube-api-access-n78st\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt\" (UID: \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:00.344945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.344934    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fkp\" (UniqueName: \"kubernetes.io/projected/02cae3ab-37b3-4947-9e3c-b37bb36318e9-kube-api-access-j9fkp\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l\" (UID: \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:00.345142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.344973    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cxh\" (UniqueName: \"kubernetes.io/projected/5925fcc5-3727-4da2-a914-b916860e8323-kube-api-access-42cxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l\" (UID: \"5925fcc5-3727-4da2-a914-b916860e8323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:00.345142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345040    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f7876-eb7d-42a8-b95b-60fecb38488c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj\" (UID: \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:00.345142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345100    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntbd4\" (UniqueName: \"kubernetes.io/projected/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-kube-api-access-ntbd4\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp\" (UID: \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:00.345142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345132    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmn87\" (UniqueName: \"kubernetes.io/projected/7833a4f4-70d2-47aa-81df-6d42a71db4ad-kube-api-access-rmn87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5\" (UID: \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:00.345308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345163    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ws68\" (UniqueName: \"kubernetes.io/projected/ab2f7876-eb7d-42a8-b95b-60fecb38488c-kube-api-access-9ws68\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj\" (UID: \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:00.345348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345321    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbd28\" (UniqueName: \"kubernetes.io/projected/aec931d7-85d3-48eb-9597-4e66cda196a1-kube-api-access-bbd28\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4\" (UID: \"aec931d7-85d3-48eb-9597-4e66cda196a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:00.345385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345355    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxrm\" (UniqueName: \"kubernetes.io/projected/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-kube-api-access-plxrm\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5\" (UID: \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:00.345416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345401    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925fcc5-3727-4da2-a914-b916860e8323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l\" (UID: \"5925fcc5-3727-4da2-a914-b916860e8323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:00.345450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345439    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59nn5\" (UniqueName: \"kubernetes.io/projected/811ad8ba-e719-4481-a4ae-a13bf468de62-kube-api-access-59nn5\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9\" (UID: \"811ad8ba-e719-4481-a4ae-a13bf468de62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:00.345533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345475    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9bc499-e4f5-42ba-ad63-9a98580a7838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt\" (UID: \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:00.345533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345509    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp\" (UID: \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:00.345721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345701    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec931d7-85d3-48eb-9597-4e66cda196a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4\" (UID: \"aec931d7-85d3-48eb-9597-4e66cda196a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:00.345801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345753    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/811ad8ba-e719-4481-a4ae-a13bf468de62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9\" (UID: \"811ad8ba-e719-4481-a4ae-a13bf468de62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:00.345801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.345784    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7833a4f4-70d2-47aa-81df-6d42a71db4ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5\" (UID: \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:00.349394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.349374    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"]
Oct 30 22:30:00.356228 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.356190    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"]
Oct 30 22:30:00.356408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.356390    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:00.362177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.362141    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:00.362454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.362440    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"]
Oct 30 22:30:00.362754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.362733    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbd28\" (UniqueName: \"kubernetes.io/projected/aec931d7-85d3-48eb-9597-4e66cda196a1-kube-api-access-bbd28\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4\" (UID: \"aec931d7-85d3-48eb-9597-4e66cda196a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:00.363102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.363082    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78st\" (UniqueName: \"kubernetes.io/projected/2b9bc499-e4f5-42ba-ad63-9a98580a7838-kube-api-access-n78st\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt\" (UID: \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:00.368496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.368470    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"]
Oct 30 22:30:00.378128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.378102    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"]
Oct 30 22:30:00.382966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.382943    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:00.383590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.382994    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"]
Oct 30 22:30:00.389793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.389774    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"]
Oct 30 22:30:00.394070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.394044    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"]
Oct 30 22:30:00.401053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.401032    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"]
Oct 30 22:30:00.404847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.404823    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"]
Oct 30 22:30:00.410309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.410246    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"]
Oct 30 22:30:00.447172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447085    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnphr\" (UniqueName: \"kubernetes.io/projected/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-kube-api-access-fnphr\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr\" (UID: \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:00.447172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447136    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr\" (UID: \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:00.447403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447262    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5\" (UID: \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:00.447403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447311    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cae3ab-37b3-4947-9e3c-b37bb36318e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l\" (UID: \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:00.447403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447368    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fkp\" (UniqueName: \"kubernetes.io/projected/02cae3ab-37b3-4947-9e3c-b37bb36318e9-kube-api-access-j9fkp\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l\" (UID: \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:00.447403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447385    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cxh\" (UniqueName: \"kubernetes.io/projected/5925fcc5-3727-4da2-a914-b916860e8323-kube-api-access-42cxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l\" (UID: \"5925fcc5-3727-4da2-a914-b916860e8323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:00.447621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447520    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b60772f-0227-4978-aecf-c38efbd91a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj\" (UID: \"9b60772f-0227-4978-aecf-c38efbd91a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:00.447671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447633    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f7876-eb7d-42a8-b95b-60fecb38488c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj\" (UID: \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:00.447707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447696    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntbd4\" (UniqueName: \"kubernetes.io/projected/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-kube-api-access-ntbd4\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp\" (UID: \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:00.447750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447721    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmn87\" (UniqueName: \"kubernetes.io/projected/7833a4f4-70d2-47aa-81df-6d42a71db4ad-kube-api-access-rmn87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5\" (UID: \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:00.447785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447756    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ws68\" (UniqueName: \"kubernetes.io/projected/ab2f7876-eb7d-42a8-b95b-60fecb38488c-kube-api-access-9ws68\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj\" (UID: \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:00.447844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447816    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpsl\" (UniqueName: \"kubernetes.io/projected/9b60772f-0227-4978-aecf-c38efbd91a15-kube-api-access-qhpsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj\" (UID: \"9b60772f-0227-4978-aecf-c38efbd91a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:00.447929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447881    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxrm\" (UniqueName: \"kubernetes.io/projected/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-kube-api-access-plxrm\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5\" (UID: \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:00.447967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447928    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925fcc5-3727-4da2-a914-b916860e8323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l\" (UID: \"5925fcc5-3727-4da2-a914-b916860e8323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:00.447998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.447967    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59nn5\" (UniqueName: \"kubernetes.io/projected/811ad8ba-e719-4481-a4ae-a13bf468de62-kube-api-access-59nn5\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9\" (UID: \"811ad8ba-e719-4481-a4ae-a13bf468de62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:00.448037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448000    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5c5\" (UniqueName: \"kubernetes.io/projected/5dc6bbad-d0d8-4154-ae36-416ace169faa-kube-api-access-8t5c5\") pod \"maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj\" (UID: \"5dc6bbad-d0d8-4154-ae36-416ace169faa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:00.448084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448068    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp\" (UID: \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:00.448126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448083    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cae3ab-37b3-4947-9e3c-b37bb36318e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l\" (UID: \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:00.448126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448083    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5\" (UID: \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:00.448351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448330    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc6bbad-d0d8-4154-ae36-416ace169faa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj\" (UID: \"5dc6bbad-d0d8-4154-ae36-416ace169faa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:00.448409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448389    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/811ad8ba-e719-4481-a4ae-a13bf468de62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9\" (UID: \"811ad8ba-e719-4481-a4ae-a13bf468de62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:00.448484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448435    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f7876-eb7d-42a8-b95b-60fecb38488c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj\" (UID: \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:00.448544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448471    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7833a4f4-70d2-47aa-81df-6d42a71db4ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5\" (UID: \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:00.448743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448726    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp\" (UID: \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:00.448885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.448866    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925fcc5-3727-4da2-a914-b916860e8323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l\" (UID: \"5925fcc5-3727-4da2-a914-b916860e8323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:00.449130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.449105    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/811ad8ba-e719-4481-a4ae-a13bf468de62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9\" (UID: \"811ad8ba-e719-4481-a4ae-a13bf468de62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:00.449517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.449495    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7833a4f4-70d2-47aa-81df-6d42a71db4ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5\" (UID: \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:00.471018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.469438    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxrm\" (UniqueName: \"kubernetes.io/projected/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-kube-api-access-plxrm\") pod \"maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5\" (UID: \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:00.472598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.471618    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cxh\" (UniqueName: \"kubernetes.io/projected/5925fcc5-3727-4da2-a914-b916860e8323-kube-api-access-42cxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l\" (UID: \"5925fcc5-3727-4da2-a914-b916860e8323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:00.472598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.471745    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59nn5\" (UniqueName: \"kubernetes.io/projected/811ad8ba-e719-4481-a4ae-a13bf468de62-kube-api-access-59nn5\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9\" (UID: \"811ad8ba-e719-4481-a4ae-a13bf468de62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:00.472598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.471837    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntbd4\" (UniqueName: \"kubernetes.io/projected/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-kube-api-access-ntbd4\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp\" (UID: \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:00.473081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.473056    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmn87\" (UniqueName: \"kubernetes.io/projected/7833a4f4-70d2-47aa-81df-6d42a71db4ad-kube-api-access-rmn87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5\" (UID: \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:00.473473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.473445    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ws68\" (UniqueName: \"kubernetes.io/projected/ab2f7876-eb7d-42a8-b95b-60fecb38488c-kube-api-access-9ws68\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj\" (UID: \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:00.474944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.474919    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fkp\" (UniqueName: \"kubernetes.io/projected/02cae3ab-37b3-4947-9e3c-b37bb36318e9-kube-api-access-j9fkp\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l\" (UID: \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:00.518969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.518488    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:00.527691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.525845    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:00.554263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.552570    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc6bbad-d0d8-4154-ae36-416ace169faa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj\" (UID: \"5dc6bbad-d0d8-4154-ae36-416ace169faa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:00.554263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.552675    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnphr\" (UniqueName: \"kubernetes.io/projected/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-kube-api-access-fnphr\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr\" (UID: \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:00.554263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.552728    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr\" (UID: \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:00.554263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.552990    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b60772f-0227-4978-aecf-c38efbd91a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj\" (UID: \"9b60772f-0227-4978-aecf-c38efbd91a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:00.554263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.553157    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpsl\" (UniqueName: \"kubernetes.io/projected/9b60772f-0227-4978-aecf-c38efbd91a15-kube-api-access-qhpsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj\" (UID: \"9b60772f-0227-4978-aecf-c38efbd91a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:00.554263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.553280    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5c5\" (UniqueName: \"kubernetes.io/projected/5dc6bbad-d0d8-4154-ae36-416ace169faa-kube-api-access-8t5c5\") pod \"maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj\" (UID: \"5dc6bbad-d0d8-4154-ae36-416ace169faa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:00.555046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.555023    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc6bbad-d0d8-4154-ae36-416ace169faa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj\" (UID: \"5dc6bbad-d0d8-4154-ae36-416ace169faa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:00.556297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.556274    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr\" (UID: \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:00.558166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.557185    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b60772f-0227-4978-aecf-c38efbd91a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj\" (UID: \"9b60772f-0227-4978-aecf-c38efbd91a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:00.585793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.585718    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnphr\" (UniqueName: \"kubernetes.io/projected/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-kube-api-access-fnphr\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr\" (UID: \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:00.591343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.591311    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpsl\" (UniqueName: \"kubernetes.io/projected/9b60772f-0227-4978-aecf-c38efbd91a15-kube-api-access-qhpsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj\" (UID: \"9b60772f-0227-4978-aecf-c38efbd91a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:00.593977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.593733    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:00.602112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.602082    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5c5\" (UniqueName: \"kubernetes.io/projected/5dc6bbad-d0d8-4154-ae36-416ace169faa-kube-api-access-8t5c5\") pod \"maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj\" (UID: \"5dc6bbad-d0d8-4154-ae36-416ace169faa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:00.616044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.614741    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:00.624787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.623171    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:00.631809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.631783    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:00.645430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.643464    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:00.648231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.648180    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:00.656361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.656192    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:00.700720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.699854    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:00.716246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.715100    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:00.716246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:00.716096    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:01.136751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.136720    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"]
Oct 30 22:30:01.139771 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:01.139740    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec931d7_85d3_48eb_9597_4e66cda196a1.slice/crio-a551150d0dcd32be4a7655aaa5ea015b5ae63c217452f03482f1465a5b66df49 WatchSource:0}: Error finding container a551150d0dcd32be4a7655aaa5ea015b5ae63c217452f03482f1465a5b66df49: Status 404 returned error can't find the container with id a551150d0dcd32be4a7655aaa5ea015b5ae63c217452f03482f1465a5b66df49
Oct 30 22:30:01.350403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.350301    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"]
Oct 30 22:30:01.389746 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.389713    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"]
Oct 30 22:30:01.390596 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:01.390569    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cae3ab_37b3_4947_9e3c_b37bb36318e9.slice/crio-165b2e0936590521b6ecba678f4a60476109067505eb22fea30f07406e2b4e26 WatchSource:0}: Error finding container 165b2e0936590521b6ecba678f4a60476109067505eb22fea30f07406e2b4e26: Status 404 returned error can't find the container with id 165b2e0936590521b6ecba678f4a60476109067505eb22fea30f07406e2b4e26
Oct 30 22:30:01.432291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.431968    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"]
Oct 30 22:30:01.436273 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:01.436184    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7833a4f4_70d2_47aa_81df_6d42a71db4ad.slice/crio-4c9d626b5792503e59d43b3d35f998f1f6eb0404ba8531186e291c7b3533a018 WatchSource:0}: Error finding container 4c9d626b5792503e59d43b3d35f998f1f6eb0404ba8531186e291c7b3533a018: Status 404 returned error can't find the container with id 4c9d626b5792503e59d43b3d35f998f1f6eb0404ba8531186e291c7b3533a018
Oct 30 22:30:01.440877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.440850    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"]
Oct 30 22:30:01.456012 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:01.455983    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2f7876_eb7d_42a8_b95b_60fecb38488c.slice/crio-e2a08cc4d2799188fea3cc8244247754880faf2e498ff5a37d4706e43dd3e1fc WatchSource:0}: Error finding container e2a08cc4d2799188fea3cc8244247754880faf2e498ff5a37d4706e43dd3e1fc: Status 404 returned error can't find the container with id e2a08cc4d2799188fea3cc8244247754880faf2e498ff5a37d4706e43dd3e1fc
Oct 30 22:30:01.469853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.469820    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"]
Oct 30 22:30:01.479731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.479703    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"]
Oct 30 22:30:01.485812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.485780    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"]
Oct 30 22:30:01.504407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.504374    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"]
Oct 30 22:30:01.512363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.512327    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"]
Oct 30 22:30:01.529100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.529073    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"]
Oct 30 22:30:01.567381 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.566956    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"]
Oct 30 22:30:01.920809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.920770    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" event={"ID":"2c8eb378-fa1e-4cce-9732-d67f5372f7e6","Type":"ContainerStarted","Data":"8b6ac6f9b145ea5645de3b92ff96152110f11f3c456d5521593f04730bac2a38"}
Oct 30 22:30:01.922263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.922199    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" event={"ID":"5925fcc5-3727-4da2-a914-b916860e8323","Type":"ContainerStarted","Data":"6cf3ddf4b128751b20356be5454ee9e90b7a660a76c305585f555769811a2ca3"}
Oct 30 22:30:01.923562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.923539    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj" event={"ID":"5dc6bbad-d0d8-4154-ae36-416ace169faa","Type":"ContainerStarted","Data":"7fc241ca25a3f62a46b89df6c2ed8f5abfa388fe564c3104f8fd99fe9c9c58ca"}
Oct 30 22:30:01.924860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.924832    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" event={"ID":"811ad8ba-e719-4481-a4ae-a13bf468de62","Type":"ContainerStarted","Data":"97e3ed52983b0e974ddae11984a754005fcd7ff776ce0543fb6e1b3d7fab8d20"}
Oct 30 22:30:01.926236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.926198    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" event={"ID":"7833a4f4-70d2-47aa-81df-6d42a71db4ad","Type":"ContainerStarted","Data":"4c9d626b5792503e59d43b3d35f998f1f6eb0404ba8531186e291c7b3533a018"}
Oct 30 22:30:01.927525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.927503    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" event={"ID":"9b60772f-0227-4978-aecf-c38efbd91a15","Type":"ContainerStarted","Data":"78960efb68754f2d29401fecf2079667421646d2dfd706f336e02b0989a403e6"}
Oct 30 22:30:01.928769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.928749    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" event={"ID":"aec3f64c-7c9b-4f83-8b0d-07dd337224b6","Type":"ContainerStarted","Data":"5407bf33dbfea670d8626e53cbad12316a484b0535e23044b525a433ca454bce"}
Oct 30 22:30:01.930052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.930027    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" event={"ID":"aec931d7-85d3-48eb-9597-4e66cda196a1","Type":"ContainerStarted","Data":"a551150d0dcd32be4a7655aaa5ea015b5ae63c217452f03482f1465a5b66df49"}
Oct 30 22:30:01.931261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.931240    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" event={"ID":"a3ce3beb-2566-4e01-92b0-02ebc5048a5d","Type":"ContainerStarted","Data":"80655847a9793893788ad3a95073a3d97036d2331a1fc2e585e00d4585e542f3"}
Oct 30 22:30:01.932546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.932516    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" event={"ID":"ab2f7876-eb7d-42a8-b95b-60fecb38488c","Type":"ContainerStarted","Data":"e2a08cc4d2799188fea3cc8244247754880faf2e498ff5a37d4706e43dd3e1fc"}
Oct 30 22:30:01.933874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.933853    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt" event={"ID":"2b9bc499-e4f5-42ba-ad63-9a98580a7838","Type":"ContainerStarted","Data":"381993cc9624fd8dae477dda1d1b08aacb5f89f0dffce4a24fa570afc5e40bb6"}
Oct 30 22:30:01.935141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:01.935123    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" event={"ID":"02cae3ab-37b3-4947-9e3c-b37bb36318e9","Type":"ContainerStarted","Data":"165b2e0936590521b6ecba678f4a60476109067505eb22fea30f07406e2b4e26"}
Oct 30 22:30:02.100362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.100324    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9ce949-0e67-4b71-8977-94cd796853cb" path="/var/lib/kubelet/pods/ec9ce949-0e67-4b71-8977-94cd796853cb/volumes"
Oct 30 22:30:02.949369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.949334    4581 generic.go:356] "Generic (PLEG): container finished" podID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerID="081dd2ff1c7d3496a00322a41c82e1e83b8b58fb9e7f45dca2552235b4105629" exitCode=60
Oct 30 22:30:02.949841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.949462    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" event={"ID":"02cae3ab-37b3-4947-9e3c-b37bb36318e9","Type":"ContainerDied","Data":"081dd2ff1c7d3496a00322a41c82e1e83b8b58fb9e7f45dca2552235b4105629"}
Oct 30 22:30:02.949841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.949792    4581 scope.go:122] "RemoveContainer" containerID="081dd2ff1c7d3496a00322a41c82e1e83b8b58fb9e7f45dca2552235b4105629"
Oct 30 22:30:02.953452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.952896    4581 generic.go:356] "Generic (PLEG): container finished" podID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerID="fe7560d4360e28604605462519b1bdfa43092915e6bfcf7c7e3672c96cd917f9" exitCode=60
Oct 30 22:30:02.953452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.952999    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" event={"ID":"2c8eb378-fa1e-4cce-9732-d67f5372f7e6","Type":"ContainerDied","Data":"fe7560d4360e28604605462519b1bdfa43092915e6bfcf7c7e3672c96cd917f9"}
Oct 30 22:30:02.953573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.953543    4581 scope.go:122] "RemoveContainer" containerID="fe7560d4360e28604605462519b1bdfa43092915e6bfcf7c7e3672c96cd917f9"
Oct 30 22:30:02.957645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.957368    4581 generic.go:356] "Generic (PLEG): container finished" podID="5925fcc5-3727-4da2-a914-b916860e8323" containerID="430eef1d5fcca9065528ff84cefc2437bf1cb30abcd0c3b62a7459038527724b" exitCode=60
Oct 30 22:30:02.957645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.957552    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" event={"ID":"5925fcc5-3727-4da2-a914-b916860e8323","Type":"ContainerDied","Data":"430eef1d5fcca9065528ff84cefc2437bf1cb30abcd0c3b62a7459038527724b"}
Oct 30 22:30:02.958744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.958287    4581 scope.go:122] "RemoveContainer" containerID="430eef1d5fcca9065528ff84cefc2437bf1cb30abcd0c3b62a7459038527724b"
Oct 30 22:30:02.963697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.963344    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj" event={"ID":"5dc6bbad-d0d8-4154-ae36-416ace169faa","Type":"ContainerStarted","Data":"ad73d556d7275e0a283595faa4a17804f9abd6e14479f57f8dce6abcd95bf2d1"}
Oct 30 22:30:02.966060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.966033    4581 generic.go:356] "Generic (PLEG): container finished" podID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerID="bf8508a8b97ad13e5d253bfee120cb264be02c83e7ac439f52ea90619c8e4ecf" exitCode=60
Oct 30 22:30:02.966145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.966121    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" event={"ID":"811ad8ba-e719-4481-a4ae-a13bf468de62","Type":"ContainerDied","Data":"bf8508a8b97ad13e5d253bfee120cb264be02c83e7ac439f52ea90619c8e4ecf"}
Oct 30 22:30:02.967336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.967310    4581 scope.go:122] "RemoveContainer" containerID="bf8508a8b97ad13e5d253bfee120cb264be02c83e7ac439f52ea90619c8e4ecf"
Oct 30 22:30:02.968617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.968591    4581 generic.go:356] "Generic (PLEG): container finished" podID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerID="f68c9b098e1b547f49351e87dddc269d33edcb35ede21e696067be76f7ecf466" exitCode=60
Oct 30 22:30:02.968903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.968844    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" event={"ID":"7833a4f4-70d2-47aa-81df-6d42a71db4ad","Type":"ContainerDied","Data":"f68c9b098e1b547f49351e87dddc269d33edcb35ede21e696067be76f7ecf466"}
Oct 30 22:30:02.969013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.968996    4581 scope.go:122] "RemoveContainer" containerID="f68c9b098e1b547f49351e87dddc269d33edcb35ede21e696067be76f7ecf466"
Oct 30 22:30:02.970988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.970962    4581 generic.go:356] "Generic (PLEG): container finished" podID="9b60772f-0227-4978-aecf-c38efbd91a15" containerID="fd3acba94f3a0c43e6f2c28633cc486093356ae2ba012fab4f326443f4eabdc2" exitCode=7
Oct 30 22:30:02.971080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.971039    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" event={"ID":"9b60772f-0227-4978-aecf-c38efbd91a15","Type":"ContainerDied","Data":"fd3acba94f3a0c43e6f2c28633cc486093356ae2ba012fab4f326443f4eabdc2"}
Oct 30 22:30:02.971348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.971329    4581 scope.go:122] "RemoveContainer" containerID="fd3acba94f3a0c43e6f2c28633cc486093356ae2ba012fab4f326443f4eabdc2"
Oct 30 22:30:02.991659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.991623    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" event={"ID":"aec3f64c-7c9b-4f83-8b0d-07dd337224b6","Type":"ContainerStarted","Data":"5263cfd3b064430da5cc6c11860ffc1edc7712fcf9e5be28ea4d6aafbe01f7db"}
Oct 30 22:30:02.992070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.992048    4581 scope.go:122] "RemoveContainer" containerID="5263cfd3b064430da5cc6c11860ffc1edc7712fcf9e5be28ea4d6aafbe01f7db"
Oct 30 22:30:02.994796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.994772    4581 generic.go:356] "Generic (PLEG): container finished" podID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerID="a24112f65c591f2e63d1d2ad043ad29acd3e627fc280a3ced7ae43d56110a8b1" exitCode=60
Oct 30 22:30:02.994891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.994845    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" event={"ID":"aec931d7-85d3-48eb-9597-4e66cda196a1","Type":"ContainerDied","Data":"a24112f65c591f2e63d1d2ad043ad29acd3e627fc280a3ced7ae43d56110a8b1"}
Oct 30 22:30:02.995261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.995239    4581 scope.go:122] "RemoveContainer" containerID="a24112f65c591f2e63d1d2ad043ad29acd3e627fc280a3ced7ae43d56110a8b1"
Oct 30 22:30:02.999925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.997683    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" event={"ID":"a3ce3beb-2566-4e01-92b0-02ebc5048a5d","Type":"ContainerStarted","Data":"62a3499f3dd80a0c88a2de9ea6b485c28792581aa8dcf8a18c225a09c62d3fbd"}
Oct 30 22:30:02.999925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:02.998068    4581 scope.go:122] "RemoveContainer" containerID="62a3499f3dd80a0c88a2de9ea6b485c28792581aa8dcf8a18c225a09c62d3fbd"
Oct 30 22:30:03.001648 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:03.001619    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" event={"ID":"ab2f7876-eb7d-42a8-b95b-60fecb38488c","Type":"ContainerStarted","Data":"854af10d2e2394696bb883aab6a4e715a6fddaf3ffb776fe9f02993302f0e2ff"}
Oct 30 22:30:03.002030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:03.002009    4581 scope.go:122] "RemoveContainer" containerID="854af10d2e2394696bb883aab6a4e715a6fddaf3ffb776fe9f02993302f0e2ff"
Oct 30 22:30:03.003935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:03.003911    4581 generic.go:356] "Generic (PLEG): container finished" podID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerID="c393afb4a0e0a892ba4d583be169c91c091be9eacb4d99eec8a61f76d478dcb6" exitCode=60
Oct 30 22:30:03.004033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:03.003997    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt" event={"ID":"2b9bc499-e4f5-42ba-ad63-9a98580a7838","Type":"ContainerDied","Data":"c393afb4a0e0a892ba4d583be169c91c091be9eacb4d99eec8a61f76d478dcb6"}
Oct 30 22:30:03.004292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:03.004273    4581 scope.go:122] "RemoveContainer" containerID="c393afb4a0e0a892ba4d583be169c91c091be9eacb4d99eec8a61f76d478dcb6"
Oct 30 22:30:03.023307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:03.023245    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj" podStartSLOduration=3.023208143 podStartE2EDuration="3.023208143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.021960551 +0000 UTC m=+1387.060899440" watchObservedRunningTime="2026-10-30 22:30:03.023208143 +0000 UTC m=+1387.062147007"
Oct 30 22:30:04.016385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.016278    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" event={"ID":"02cae3ab-37b3-4947-9e3c-b37bb36318e9","Type":"ContainerStarted","Data":"8abdab36cd39057ed2440ee3b1b4711e414c7f271a2a1bc40c517e4ec233341b"}
Oct 30 22:30:04.023543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.023504    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" event={"ID":"2c8eb378-fa1e-4cce-9732-d67f5372f7e6","Type":"ContainerStarted","Data":"0000886d2b398de8c677337254f20fc28549c60edfc49a4237a7616ad50870da"}
Oct 30 22:30:04.026001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.025961    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" event={"ID":"5925fcc5-3727-4da2-a914-b916860e8323","Type":"ContainerStarted","Data":"7f754eea07e89b9f1f9960792da18b1e490f7d084e4fcef74e784cc81c686d27"}
Oct 30 22:30:04.029053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.029026    4581 generic.go:356] "Generic (PLEG): container finished" podID="5dc6bbad-d0d8-4154-ae36-416ace169faa" containerID="ad73d556d7275e0a283595faa4a17804f9abd6e14479f57f8dce6abcd95bf2d1" exitCode=60
Oct 30 22:30:04.029183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.029092    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj" event={"ID":"5dc6bbad-d0d8-4154-ae36-416ace169faa","Type":"ContainerDied","Data":"ad73d556d7275e0a283595faa4a17804f9abd6e14479f57f8dce6abcd95bf2d1"}
Oct 30 22:30:04.029378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.029361    4581 scope.go:122] "RemoveContainer" containerID="ad73d556d7275e0a283595faa4a17804f9abd6e14479f57f8dce6abcd95bf2d1"
Oct 30 22:30:04.034196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.034168    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" event={"ID":"811ad8ba-e719-4481-a4ae-a13bf468de62","Type":"ContainerStarted","Data":"00daeb20ba7fd05e9dc032e033aaede7c21c0522107ad8b5286e475de424b8c5"}
Oct 30 22:30:04.041823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.041786    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" event={"ID":"7833a4f4-70d2-47aa-81df-6d42a71db4ad","Type":"ContainerStarted","Data":"401bb5a447d69103124fa8f1739dfdb5cfd3519842bb8ace4afe705591d53849"}
Oct 30 22:30:04.044002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.043973    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" event={"ID":"9b60772f-0227-4978-aecf-c38efbd91a15","Type":"ContainerStarted","Data":"39ae108288e0a0fe9753309cef5844b65525a6666b945d2d50e59a25d641e2ea"}
Oct 30 22:30:04.046018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.045988    4581 generic.go:356] "Generic (PLEG): container finished" podID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerID="5263cfd3b064430da5cc6c11860ffc1edc7712fcf9e5be28ea4d6aafbe01f7db" exitCode=60
Oct 30 22:30:04.046128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.046076    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" event={"ID":"aec3f64c-7c9b-4f83-8b0d-07dd337224b6","Type":"ContainerDied","Data":"5263cfd3b064430da5cc6c11860ffc1edc7712fcf9e5be28ea4d6aafbe01f7db"}
Oct 30 22:30:04.046210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.046126    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" event={"ID":"aec3f64c-7c9b-4f83-8b0d-07dd337224b6","Type":"ContainerStarted","Data":"2ce610877ea331127febf8372f38bed107f97df0c20a9fadd39c625119a85e82"}
Oct 30 22:30:04.048150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.048127    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" event={"ID":"aec931d7-85d3-48eb-9597-4e66cda196a1","Type":"ContainerStarted","Data":"817ec8dd1dc3800b2126d99ee3f35eb8e3e41b68ea740f23451e1d38e029488e"}
Oct 30 22:30:04.050085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.050057    4581 generic.go:356] "Generic (PLEG): container finished" podID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerID="62a3499f3dd80a0c88a2de9ea6b485c28792581aa8dcf8a18c225a09c62d3fbd" exitCode=60
Oct 30 22:30:04.050200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.050132    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" event={"ID":"a3ce3beb-2566-4e01-92b0-02ebc5048a5d","Type":"ContainerDied","Data":"62a3499f3dd80a0c88a2de9ea6b485c28792581aa8dcf8a18c225a09c62d3fbd"}
Oct 30 22:30:04.050200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.050174    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" event={"ID":"a3ce3beb-2566-4e01-92b0-02ebc5048a5d","Type":"ContainerStarted","Data":"bc52102c9c295336c9230c70ca2f74cf5a36d3170ccbfb90c4cacda10a4d9132"}
Oct 30 22:30:04.062640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.062614    4581 generic.go:356] "Generic (PLEG): container finished" podID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerID="854af10d2e2394696bb883aab6a4e715a6fddaf3ffb776fe9f02993302f0e2ff" exitCode=60
Oct 30 22:30:04.062802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.062664    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" event={"ID":"ab2f7876-eb7d-42a8-b95b-60fecb38488c","Type":"ContainerDied","Data":"854af10d2e2394696bb883aab6a4e715a6fddaf3ffb776fe9f02993302f0e2ff"}
Oct 30 22:30:04.066493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.066437    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" podStartSLOduration=4.066418266 podStartE2EDuration="4.066418266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.060719204 +0000 UTC m=+1388.099658093" watchObservedRunningTime="2026-10-30 22:30:04.066418266 +0000 UTC m=+1388.105357154"
Oct 30 22:30:04.105796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.105741    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" podStartSLOduration=4.105725757 podStartE2EDuration="4.105725757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.100034854 +0000 UTC m=+1388.138973743" watchObservedRunningTime="2026-10-30 22:30:04.105725757 +0000 UTC m=+1388.144664643"
Oct 30 22:30:04.136879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.136805    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" podStartSLOduration=4.136791157 podStartE2EDuration="4.136791157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.133063962 +0000 UTC m=+1388.172002847" watchObservedRunningTime="2026-10-30 22:30:04.136791157 +0000 UTC m=+1388.175730043"
Oct 30 22:30:04.171603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.171541    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" podStartSLOduration=4.171522427 podStartE2EDuration="4.171522427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.16914981 +0000 UTC m=+1388.208088695" watchObservedRunningTime="2026-10-30 22:30:04.171522427 +0000 UTC m=+1388.210461314"
Oct 30 22:30:04.205424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.205371    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" podStartSLOduration=4.20535728 podStartE2EDuration="4.20535728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.204546213 +0000 UTC m=+1388.243485098" watchObservedRunningTime="2026-10-30 22:30:04.20535728 +0000 UTC m=+1388.244296160"
Oct 30 22:30:04.242900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.242842    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" podStartSLOduration=4.242827161 podStartE2EDuration="4.242827161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.237397608 +0000 UTC m=+1388.276336493" watchObservedRunningTime="2026-10-30 22:30:04.242827161 +0000 UTC m=+1388.281766082"
Oct 30 22:30:04.273292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.273061    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" podStartSLOduration=4.273040584 podStartE2EDuration="4.273040584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.271770053 +0000 UTC m=+1388.310708943" watchObservedRunningTime="2026-10-30 22:30:04.273040584 +0000 UTC m=+1388.311979472"
Oct 30 22:30:04.311100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.311036    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" podStartSLOduration=4.311019945 podStartE2EDuration="4.311019945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.305571949 +0000 UTC m=+1388.344510837" watchObservedRunningTime="2026-10-30 22:30:04.311019945 +0000 UTC m=+1388.349958831"
Oct 30 22:30:04.339536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:04.339469    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" podStartSLOduration=4.339449752 podStartE2EDuration="4.339449752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.339111075 +0000 UTC m=+1388.378049958" watchObservedRunningTime="2026-10-30 22:30:04.339449752 +0000 UTC m=+1388.378388639"
Oct 30 22:30:05.074951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.074912    4581 generic.go:356] "Generic (PLEG): container finished" podID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerID="2ce610877ea331127febf8372f38bed107f97df0c20a9fadd39c625119a85e82" exitCode=60
Oct 30 22:30:05.075474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.074995    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" event={"ID":"aec3f64c-7c9b-4f83-8b0d-07dd337224b6","Type":"ContainerDied","Data":"2ce610877ea331127febf8372f38bed107f97df0c20a9fadd39c625119a85e82"}
Oct 30 22:30:05.075474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.075105    4581 scope.go:122] "RemoveContainer" containerID="5263cfd3b064430da5cc6c11860ffc1edc7712fcf9e5be28ea4d6aafbe01f7db"
Oct 30 22:30:05.075642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.075477    4581 scope.go:122] "RemoveContainer" containerID="2ce610877ea331127febf8372f38bed107f97df0c20a9fadd39c625119a85e82"
Oct 30 22:30:05.075936 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.075905    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5_odh-ai-gateway-infra(aec3f64c-7c9b-4f83-8b0d-07dd337224b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6"
Oct 30 22:30:05.077543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.077521    4581 generic.go:356] "Generic (PLEG): container finished" podID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerID="817ec8dd1dc3800b2126d99ee3f35eb8e3e41b68ea740f23451e1d38e029488e" exitCode=60
Oct 30 22:30:05.077648 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.077598    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" event={"ID":"aec931d7-85d3-48eb-9597-4e66cda196a1","Type":"ContainerDied","Data":"817ec8dd1dc3800b2126d99ee3f35eb8e3e41b68ea740f23451e1d38e029488e"}
Oct 30 22:30:05.077948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.077916    4581 scope.go:122] "RemoveContainer" containerID="817ec8dd1dc3800b2126d99ee3f35eb8e3e41b68ea740f23451e1d38e029488e"
Oct 30 22:30:05.078369 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.078345    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4_odh-ai-gateway-infra(aec931d7-85d3-48eb-9597-4e66cda196a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1"
Oct 30 22:30:05.079747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.079726    4581 generic.go:356] "Generic (PLEG): container finished" podID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerID="bc52102c9c295336c9230c70ca2f74cf5a36d3170ccbfb90c4cacda10a4d9132" exitCode=60
Oct 30 22:30:05.079859 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.079809    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" event={"ID":"a3ce3beb-2566-4e01-92b0-02ebc5048a5d","Type":"ContainerDied","Data":"bc52102c9c295336c9230c70ca2f74cf5a36d3170ccbfb90c4cacda10a4d9132"}
Oct 30 22:30:05.080056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.080038    4581 scope.go:122] "RemoveContainer" containerID="bc52102c9c295336c9230c70ca2f74cf5a36d3170ccbfb90c4cacda10a4d9132"
Oct 30 22:30:05.080397 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.080372    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp_odh-ai-gateway-infra(a3ce3beb-2566-4e01-92b0-02ebc5048a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d"
Oct 30 22:30:05.081933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.081879    4581 generic.go:356] "Generic (PLEG): container finished" podID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerID="b096e5d53d9de95b2f16fe0aa3fff6b4c357c32a0bed5d5223fa8b9b1e3cda92" exitCode=60
Oct 30 22:30:05.082027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.081968    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" event={"ID":"ab2f7876-eb7d-42a8-b95b-60fecb38488c","Type":"ContainerDied","Data":"b096e5d53d9de95b2f16fe0aa3fff6b4c357c32a0bed5d5223fa8b9b1e3cda92"}
Oct 30 22:30:05.082268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.082250    4581 scope.go:122] "RemoveContainer" containerID="b096e5d53d9de95b2f16fe0aa3fff6b4c357c32a0bed5d5223fa8b9b1e3cda92"
Oct 30 22:30:05.082591 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.082570    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj_odh-ai-gateway-infra(ab2f7876-eb7d-42a8-b95b-60fecb38488c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c"
Oct 30 22:30:05.084393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.084372    4581 generic.go:356] "Generic (PLEG): container finished" podID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerID="fec79c692dddf8139df66b3f1a774f47bdba6a29ad6aeb91d9a88106cc969b0f" exitCode=60
Oct 30 22:30:05.084493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.084458    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt" event={"ID":"2b9bc499-e4f5-42ba-ad63-9a98580a7838","Type":"ContainerDied","Data":"fec79c692dddf8139df66b3f1a774f47bdba6a29ad6aeb91d9a88106cc969b0f"}
Oct 30 22:30:05.084735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.084718    4581 scope.go:122] "RemoveContainer" containerID="fec79c692dddf8139df66b3f1a774f47bdba6a29ad6aeb91d9a88106cc969b0f"
Oct 30 22:30:05.085056 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.085035    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt_odh-ai-gateway-infra(2b9bc499-e4f5-42ba-ad63-9a98580a7838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838"
Oct 30 22:30:05.086797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.086700    4581 generic.go:356] "Generic (PLEG): container finished" podID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerID="8abdab36cd39057ed2440ee3b1b4711e414c7f271a2a1bc40c517e4ec233341b" exitCode=60
Oct 30 22:30:05.086797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.086734    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" event={"ID":"02cae3ab-37b3-4947-9e3c-b37bb36318e9","Type":"ContainerDied","Data":"8abdab36cd39057ed2440ee3b1b4711e414c7f271a2a1bc40c517e4ec233341b"}
Oct 30 22:30:05.087083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.087067    4581 scope.go:122] "RemoveContainer" containerID="8abdab36cd39057ed2440ee3b1b4711e414c7f271a2a1bc40c517e4ec233341b"
Oct 30 22:30:05.087441 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.087418    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l_odh-ai-gateway-infra(02cae3ab-37b3-4947-9e3c-b37bb36318e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9"
Oct 30 22:30:05.089110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.089088    4581 generic.go:356] "Generic (PLEG): container finished" podID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerID="0000886d2b398de8c677337254f20fc28549c60edfc49a4237a7616ad50870da" exitCode=60
Oct 30 22:30:05.089227 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.089133    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" event={"ID":"2c8eb378-fa1e-4cce-9732-d67f5372f7e6","Type":"ContainerDied","Data":"0000886d2b398de8c677337254f20fc28549c60edfc49a4237a7616ad50870da"}
Oct 30 22:30:05.089399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.089381    4581 scope.go:122] "RemoveContainer" containerID="0000886d2b398de8c677337254f20fc28549c60edfc49a4237a7616ad50870da"
Oct 30 22:30:05.089698 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.089661    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr_odh-ai-gateway-infra(2c8eb378-fa1e-4cce-9732-d67f5372f7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6"
Oct 30 22:30:05.091308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.091287    4581 generic.go:356] "Generic (PLEG): container finished" podID="5925fcc5-3727-4da2-a914-b916860e8323" containerID="7f754eea07e89b9f1f9960792da18b1e490f7d084e4fcef74e784cc81c686d27" exitCode=60
Oct 30 22:30:05.091415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.091366    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" event={"ID":"5925fcc5-3727-4da2-a914-b916860e8323","Type":"ContainerDied","Data":"7f754eea07e89b9f1f9960792da18b1e490f7d084e4fcef74e784cc81c686d27"}
Oct 30 22:30:05.091597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.091580    4581 scope.go:122] "RemoveContainer" containerID="7f754eea07e89b9f1f9960792da18b1e490f7d084e4fcef74e784cc81c686d27"
Oct 30 22:30:05.091896 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.091869    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l_odh-ai-gateway-infra(5925fcc5-3727-4da2-a914-b916860e8323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" podUID="5925fcc5-3727-4da2-a914-b916860e8323"
Oct 30 22:30:05.093567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.093545    4581 generic.go:356] "Generic (PLEG): container finished" podID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerID="00daeb20ba7fd05e9dc032e033aaede7c21c0522107ad8b5286e475de424b8c5" exitCode=60
Oct 30 22:30:05.093786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.093766    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" event={"ID":"811ad8ba-e719-4481-a4ae-a13bf468de62","Type":"ContainerDied","Data":"00daeb20ba7fd05e9dc032e033aaede7c21c0522107ad8b5286e475de424b8c5"}
Oct 30 22:30:05.093911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.093898    4581 scope.go:122] "RemoveContainer" containerID="00daeb20ba7fd05e9dc032e033aaede7c21c0522107ad8b5286e475de424b8c5"
Oct 30 22:30:05.094182 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.094162    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9_odh-ai-gateway-infra(811ad8ba-e719-4481-a4ae-a13bf468de62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62"
Oct 30 22:30:05.097129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.097106    4581 generic.go:356] "Generic (PLEG): container finished" podID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerID="401bb5a447d69103124fa8f1739dfdb5cfd3519842bb8ace4afe705591d53849" exitCode=60
Oct 30 22:30:05.097211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.097193    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" event={"ID":"7833a4f4-70d2-47aa-81df-6d42a71db4ad","Type":"ContainerDied","Data":"401bb5a447d69103124fa8f1739dfdb5cfd3519842bb8ace4afe705591d53849"}
Oct 30 22:30:05.097610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.097488    4581 scope.go:122] "RemoveContainer" containerID="401bb5a447d69103124fa8f1739dfdb5cfd3519842bb8ace4afe705591d53849"
Oct 30 22:30:05.097791 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.097770    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5_odh-ai-gateway-infra(7833a4f4-70d2-47aa-81df-6d42a71db4ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad"
Oct 30 22:30:05.099186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.099166    4581 generic.go:356] "Generic (PLEG): container finished" podID="9b60772f-0227-4978-aecf-c38efbd91a15" containerID="39ae108288e0a0fe9753309cef5844b65525a6666b945d2d50e59a25d641e2ea" exitCode=7
Oct 30 22:30:05.099305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.099250    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" event={"ID":"9b60772f-0227-4978-aecf-c38efbd91a15","Type":"ContainerDied","Data":"39ae108288e0a0fe9753309cef5844b65525a6666b945d2d50e59a25d641e2ea"}
Oct 30 22:30:05.099480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.099465    4581 scope.go:122] "RemoveContainer" containerID="39ae108288e0a0fe9753309cef5844b65525a6666b945d2d50e59a25d641e2ea"
Oct 30 22:30:05.099721 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:05.099699    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj_odh-ai-gateway-infra(9b60772f-0227-4978-aecf-c38efbd91a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" podUID="9b60772f-0227-4978-aecf-c38efbd91a15"
Oct 30 22:30:05.162325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.162294    4581 scope.go:122] "RemoveContainer" containerID="a24112f65c591f2e63d1d2ad043ad29acd3e627fc280a3ced7ae43d56110a8b1"
Oct 30 22:30:05.185265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.185241    4581 scope.go:122] "RemoveContainer" containerID="62a3499f3dd80a0c88a2de9ea6b485c28792581aa8dcf8a18c225a09c62d3fbd"
Oct 30 22:30:05.201316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.201295    4581 scope.go:122] "RemoveContainer" containerID="854af10d2e2394696bb883aab6a4e715a6fddaf3ffb776fe9f02993302f0e2ff"
Oct 30 22:30:05.228331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.227939    4581 scope.go:122] "RemoveContainer" containerID="c393afb4a0e0a892ba4d583be169c91c091be9eacb4d99eec8a61f76d478dcb6"
Oct 30 22:30:05.248017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.247995    4581 scope.go:122] "RemoveContainer" containerID="081dd2ff1c7d3496a00322a41c82e1e83b8b58fb9e7f45dca2552235b4105629"
Oct 30 22:30:05.264331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.264307    4581 scope.go:122] "RemoveContainer" containerID="fe7560d4360e28604605462519b1bdfa43092915e6bfcf7c7e3672c96cd917f9"
Oct 30 22:30:05.279948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.279926    4581 scope.go:122] "RemoveContainer" containerID="430eef1d5fcca9065528ff84cefc2437bf1cb30abcd0c3b62a7459038527724b"
Oct 30 22:30:05.295360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.295341    4581 scope.go:122] "RemoveContainer" containerID="bf8508a8b97ad13e5d253bfee120cb264be02c83e7ac439f52ea90619c8e4ecf"
Oct 30 22:30:05.313747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.313720    4581 scope.go:122] "RemoveContainer" containerID="f68c9b098e1b547f49351e87dddc269d33edcb35ede21e696067be76f7ecf466"
Oct 30 22:30:05.333786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:05.332592    4581 scope.go:122] "RemoveContainer" containerID="fd3acba94f3a0c43e6f2c28633cc486093356ae2ba012fab4f326443f4eabdc2"
Oct 30 22:30:06.112060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.112035    4581 scope.go:122] "RemoveContainer" containerID="0000886d2b398de8c677337254f20fc28549c60edfc49a4237a7616ad50870da"
Oct 30 22:30:06.112614 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.112581    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr_odh-ai-gateway-infra(2c8eb378-fa1e-4cce-9732-d67f5372f7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6"
Oct 30 22:30:06.114032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.114008    4581 scope.go:122] "RemoveContainer" containerID="7f754eea07e89b9f1f9960792da18b1e490f7d084e4fcef74e784cc81c686d27"
Oct 30 22:30:06.114452 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.114425    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l_odh-ai-gateway-infra(5925fcc5-3727-4da2-a914-b916860e8323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" podUID="5925fcc5-3727-4da2-a914-b916860e8323"
Oct 30 22:30:06.116170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.116099    4581 generic.go:356] "Generic (PLEG): container finished" podID="5dc6bbad-d0d8-4154-ae36-416ace169faa" containerID="9b15560ee6a4adb573027d2fd8156f3c60fcacff06beeabdceae8e58b622d06a" exitCode=60
Oct 30 22:30:06.116327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.116174    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj" event={"ID":"5dc6bbad-d0d8-4154-ae36-416ace169faa","Type":"ContainerDied","Data":"9b15560ee6a4adb573027d2fd8156f3c60fcacff06beeabdceae8e58b622d06a"}
Oct 30 22:30:06.116327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.116242    4581 scope.go:122] "RemoveContainer" containerID="ad73d556d7275e0a283595faa4a17804f9abd6e14479f57f8dce6abcd95bf2d1"
Oct 30 22:30:06.116543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.116526    4581 scope.go:122] "RemoveContainer" containerID="9b15560ee6a4adb573027d2fd8156f3c60fcacff06beeabdceae8e58b622d06a"
Oct 30 22:30:06.116886 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.116861    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj_odh-ai-gateway-infra(5dc6bbad-d0d8-4154-ae36-416ace169faa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa"
Oct 30 22:30:06.118541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.118516    4581 scope.go:122] "RemoveContainer" containerID="00daeb20ba7fd05e9dc032e033aaede7c21c0522107ad8b5286e475de424b8c5"
Oct 30 22:30:06.118854 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.118834    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9_odh-ai-gateway-infra(811ad8ba-e719-4481-a4ae-a13bf468de62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62"
Oct 30 22:30:06.120504 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.120476    4581 scope.go:122] "RemoveContainer" containerID="401bb5a447d69103124fa8f1739dfdb5cfd3519842bb8ace4afe705591d53849"
Oct 30 22:30:06.120727 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.120702    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5_odh-ai-gateway-infra(7833a4f4-70d2-47aa-81df-6d42a71db4ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad"
Oct 30 22:30:06.122567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.122547    4581 scope.go:122] "RemoveContainer" containerID="39ae108288e0a0fe9753309cef5844b65525a6666b945d2d50e59a25d641e2ea"
Oct 30 22:30:06.122854 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.122831    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj_odh-ai-gateway-infra(9b60772f-0227-4978-aecf-c38efbd91a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" podUID="9b60772f-0227-4978-aecf-c38efbd91a15"
Oct 30 22:30:06.124478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.124458    4581 scope.go:122] "RemoveContainer" containerID="2ce610877ea331127febf8372f38bed107f97df0c20a9fadd39c625119a85e82"
Oct 30 22:30:06.124795 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.124771    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5_odh-ai-gateway-infra(aec3f64c-7c9b-4f83-8b0d-07dd337224b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6"
Oct 30 22:30:06.126768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.126748    4581 scope.go:122] "RemoveContainer" containerID="817ec8dd1dc3800b2126d99ee3f35eb8e3e41b68ea740f23451e1d38e029488e"
Oct 30 22:30:06.127036 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.127014    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4_odh-ai-gateway-infra(aec931d7-85d3-48eb-9597-4e66cda196a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1"
Oct 30 22:30:06.128493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.128474    4581 scope.go:122] "RemoveContainer" containerID="bc52102c9c295336c9230c70ca2f74cf5a36d3170ccbfb90c4cacda10a4d9132"
Oct 30 22:30:06.128730 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.128707    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp_odh-ai-gateway-infra(a3ce3beb-2566-4e01-92b0-02ebc5048a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d"
Oct 30 22:30:06.130156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.130138    4581 scope.go:122] "RemoveContainer" containerID="b096e5d53d9de95b2f16fe0aa3fff6b4c357c32a0bed5d5223fa8b9b1e3cda92"
Oct 30 22:30:06.130410 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.130386    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj_odh-ai-gateway-infra(ab2f7876-eb7d-42a8-b95b-60fecb38488c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c"
Oct 30 22:30:06.131803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.131783    4581 scope.go:122] "RemoveContainer" containerID="fec79c692dddf8139df66b3f1a774f47bdba6a29ad6aeb91d9a88106cc969b0f"
Oct 30 22:30:06.132030 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.132011    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt_odh-ai-gateway-infra(2b9bc499-e4f5-42ba-ad63-9a98580a7838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838"
Oct 30 22:30:06.133502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:06.133487    4581 scope.go:122] "RemoveContainer" containerID="8abdab36cd39057ed2440ee3b1b4711e414c7f271a2a1bc40c517e4ec233341b"
Oct 30 22:30:06.133722 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:06.133680    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l_odh-ai-gateway-infra(02cae3ab-37b3-4947-9e3c-b37bb36318e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9"
Oct 30 22:30:07.145380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:07.145344    4581 scope.go:122] "RemoveContainer" containerID="9b15560ee6a4adb573027d2fd8156f3c60fcacff06beeabdceae8e58b622d06a"
Oct 30 22:30:07.145848 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:07.145816    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj_odh-ai-gateway-infra(5dc6bbad-d0d8-4154-ae36-416ace169faa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa"
Oct 30 22:30:09.583133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:09.583085    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"]
Oct 30 22:30:09.804380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:09.804353    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:09.821832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:09.821801    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:30:09.958932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:09.958881    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc6bbad-d0d8-4154-ae36-416ace169faa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc6bbad-d0d8-4154-ae36-416ace169faa-maas-api-service-ca\") pod \"5dc6bbad-d0d8-4154-ae36-416ace169faa\" (UID: \"5dc6bbad-d0d8-4154-ae36-416ace169faa\") "
Oct 30 22:30:09.959115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:09.958949    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc6bbad-d0d8-4154-ae36-416ace169faa-kube-api-access-8t5c5\" (UniqueName: \"kubernetes.io/projected/5dc6bbad-d0d8-4154-ae36-416ace169faa-kube-api-access-8t5c5\") pod \"5dc6bbad-d0d8-4154-ae36-416ace169faa\" (UID: \"5dc6bbad-d0d8-4154-ae36-416ace169faa\") "
Oct 30 22:30:09.959766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:09.959733    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc6bbad-d0d8-4154-ae36-416ace169faa-maas-api-service-ca" pod "5dc6bbad-d0d8-4154-ae36-416ace169faa" (UID: "5dc6bbad-d0d8-4154-ae36-416ace169faa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:09.960084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:09.960054    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc6bbad-d0d8-4154-ae36-416ace169faa-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:09.963004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:09.962965    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc6bbad-d0d8-4154-ae36-416ace169faa-kube-api-access-8t5c5" pod "5dc6bbad-d0d8-4154-ae36-416ace169faa" (UID: "5dc6bbad-d0d8-4154-ae36-416ace169faa"). InnerVolumeSpecName "kube-api-access-8t5c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.062158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:10.061870    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5c5\" (UniqueName: \"kubernetes.io/projected/5dc6bbad-d0d8-4154-ae36-416ace169faa-kube-api-access-8t5c5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:10.242650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:10.242621    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"
Oct 30 22:30:10.243023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:10.242622    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj" event={"ID":"5dc6bbad-d0d8-4154-ae36-416ace169faa","Type":"ContainerDied","Data":"7fc241ca25a3f62a46b89df6c2ed8f5abfa388fe564c3104f8fd99fe9c9c58ca"}
Oct 30 22:30:10.243023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:10.242992    4581 scope.go:122] "RemoveContainer" containerID="9b15560ee6a4adb573027d2fd8156f3c60fcacff06beeabdceae8e58b622d06a"
Oct 30 22:30:10.306525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:10.306494    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"]
Oct 30 22:30:10.319994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:10.319949    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d399fb07-29889990-vxtcj"]
Oct 30 22:30:10.618705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:10.618672    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"]
Oct 30 22:30:10.624032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:10.624003    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d399fb07-sf7kt"]
Oct 30 22:30:10.824525 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:10.824487    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:30:12.099537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:12.099498    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031a4fef-a653-4c37-955f-d1d7aec77833" path="/var/lib/kubelet/pods/031a4fef-a653-4c37-955f-d1d7aec77833/volumes"
Oct 30 22:30:12.100026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:12.100014    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa" path="/var/lib/kubelet/pods/5dc6bbad-d0d8-4154-ae36-416ace169faa/volumes"
Oct 30 22:30:15.090038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:15.090006    4581 scope.go:122] "RemoveContainer" containerID="401bb5a447d69103124fa8f1739dfdb5cfd3519842bb8ace4afe705591d53849"
Oct 30 22:30:15.090787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:15.090254    4581 scope.go:122] "RemoveContainer" containerID="817ec8dd1dc3800b2126d99ee3f35eb8e3e41b68ea740f23451e1d38e029488e"
Oct 30 22:30:16.096658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:16.096616    4581 scope.go:122] "RemoveContainer" containerID="2ce610877ea331127febf8372f38bed107f97df0c20a9fadd39c625119a85e82"
Oct 30 22:30:16.097625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:16.097275    4581 scope.go:122] "RemoveContainer" containerID="39ae108288e0a0fe9753309cef5844b65525a6666b945d2d50e59a25d641e2ea"
Oct 30 22:30:16.100307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:16.097795    4581 scope.go:122] "RemoveContainer" containerID="fec79c692dddf8139df66b3f1a774f47bdba6a29ad6aeb91d9a88106cc969b0f"
Oct 30 22:30:16.318853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:16.318759    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" event={"ID":"7833a4f4-70d2-47aa-81df-6d42a71db4ad","Type":"ContainerStarted","Data":"8901c3181b5946388dc584366ad2aa6fceb070beb986b91c756d97c668e51637"}
Oct 30 22:30:16.320985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:16.320959    4581 generic.go:356] "Generic (PLEG): container finished" podID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerID="ff562dc0d0a5bbd6ed64c662d4ef6644b656b9d61820501547a173c70f061e19" exitCode=60
Oct 30 22:30:16.321127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:16.321030    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" event={"ID":"aec931d7-85d3-48eb-9597-4e66cda196a1","Type":"ContainerDied","Data":"ff562dc0d0a5bbd6ed64c662d4ef6644b656b9d61820501547a173c70f061e19"}
Oct 30 22:30:16.321127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:16.321092    4581 scope.go:122] "RemoveContainer" containerID="817ec8dd1dc3800b2126d99ee3f35eb8e3e41b68ea740f23451e1d38e029488e"
Oct 30 22:30:16.321489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:16.321473    4581 scope.go:122] "RemoveContainer" containerID="ff562dc0d0a5bbd6ed64c662d4ef6644b656b9d61820501547a173c70f061e19"
Oct 30 22:30:16.321839 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:16.321811    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4_odh-ai-gateway-infra(aec931d7-85d3-48eb-9597-4e66cda196a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1"
Oct 30 22:30:17.088993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.088892    4581 scope.go:122] "RemoveContainer" containerID="b096e5d53d9de95b2f16fe0aa3fff6b4c357c32a0bed5d5223fa8b9b1e3cda92"
Oct 30 22:30:17.089383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.089360    4581 scope.go:122] "RemoveContainer" containerID="0000886d2b398de8c677337254f20fc28549c60edfc49a4237a7616ad50870da"
Oct 30 22:30:17.089594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.089566    4581 scope.go:122] "RemoveContainer" containerID="7f754eea07e89b9f1f9960792da18b1e490f7d084e4fcef74e784cc81c686d27"
Oct 30 22:30:17.335004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.334973    4581 generic.go:356] "Generic (PLEG): container finished" podID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerID="f2f9d3bd4a591ec50eee64196011fe7853595fa261bfa5af1683bcbbc3276964" exitCode=60
Oct 30 22:30:17.335558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.335053    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt" event={"ID":"2b9bc499-e4f5-42ba-ad63-9a98580a7838","Type":"ContainerDied","Data":"f2f9d3bd4a591ec50eee64196011fe7853595fa261bfa5af1683bcbbc3276964"}
Oct 30 22:30:17.335558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.335121    4581 scope.go:122] "RemoveContainer" containerID="fec79c692dddf8139df66b3f1a774f47bdba6a29ad6aeb91d9a88106cc969b0f"
Oct 30 22:30:17.335694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.335649    4581 scope.go:122] "RemoveContainer" containerID="f2f9d3bd4a591ec50eee64196011fe7853595fa261bfa5af1683bcbbc3276964"
Oct 30 22:30:17.336157 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:17.336127    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt_odh-ai-gateway-infra(2b9bc499-e4f5-42ba-ad63-9a98580a7838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838"
Oct 30 22:30:17.338767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.338737    4581 generic.go:356] "Generic (PLEG): container finished" podID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerID="8901c3181b5946388dc584366ad2aa6fceb070beb986b91c756d97c668e51637" exitCode=60
Oct 30 22:30:17.338879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.338811    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" event={"ID":"7833a4f4-70d2-47aa-81df-6d42a71db4ad","Type":"ContainerDied","Data":"8901c3181b5946388dc584366ad2aa6fceb070beb986b91c756d97c668e51637"}
Oct 30 22:30:17.339408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.339335    4581 scope.go:122] "RemoveContainer" containerID="8901c3181b5946388dc584366ad2aa6fceb070beb986b91c756d97c668e51637"
Oct 30 22:30:17.339786 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:17.339761    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5_odh-ai-gateway-infra(7833a4f4-70d2-47aa-81df-6d42a71db4ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad"
Oct 30 22:30:17.341925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.341903    4581 generic.go:356] "Generic (PLEG): container finished" podID="9b60772f-0227-4978-aecf-c38efbd91a15" containerID="cac92f0040075bb6bbb4d89b60a83a6530703ed3a6c5e4447c142dbabe8ae633" exitCode=60
Oct 30 22:30:17.342025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.341976    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" event={"ID":"9b60772f-0227-4978-aecf-c38efbd91a15","Type":"ContainerDied","Data":"cac92f0040075bb6bbb4d89b60a83a6530703ed3a6c5e4447c142dbabe8ae633"}
Oct 30 22:30:17.342441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.342421    4581 scope.go:122] "RemoveContainer" containerID="cac92f0040075bb6bbb4d89b60a83a6530703ed3a6c5e4447c142dbabe8ae633"
Oct 30 22:30:17.342823 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:17.342798    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj_odh-ai-gateway-infra(9b60772f-0227-4978-aecf-c38efbd91a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" podUID="9b60772f-0227-4978-aecf-c38efbd91a15"
Oct 30 22:30:17.344607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.344562    4581 generic.go:356] "Generic (PLEG): container finished" podID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerID="fde35add84181dd31af5bf95df7f72472d409232c793af1df136c3a357721a50" exitCode=60
Oct 30 22:30:17.344707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.344612    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" event={"ID":"aec3f64c-7c9b-4f83-8b0d-07dd337224b6","Type":"ContainerDied","Data":"fde35add84181dd31af5bf95df7f72472d409232c793af1df136c3a357721a50"}
Oct 30 22:30:17.345021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.345002    4581 scope.go:122] "RemoveContainer" containerID="fde35add84181dd31af5bf95df7f72472d409232c793af1df136c3a357721a50"
Oct 30 22:30:17.345428 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:17.345408    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5_odh-ai-gateway-infra(aec3f64c-7c9b-4f83-8b0d-07dd337224b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6"
Oct 30 22:30:17.369667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.369638    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"]
Oct 30 22:30:17.393339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.393315    4581 scope.go:122] "RemoveContainer" containerID="401bb5a447d69103124fa8f1739dfdb5cfd3519842bb8ace4afe705591d53849"
Oct 30 22:30:17.408614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.408588    4581 scope.go:122] "RemoveContainer" containerID="39ae108288e0a0fe9753309cef5844b65525a6666b945d2d50e59a25d641e2ea"
Oct 30 22:30:17.424630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.424601    4581 scope.go:122] "RemoveContainer" containerID="2ce610877ea331127febf8372f38bed107f97df0c20a9fadd39c625119a85e82"
Oct 30 22:30:17.433696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.433667    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"]
Oct 30 22:30:17.638930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.638903    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:17.714637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.714600    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec931d7-85d3-48eb-9597-4e66cda196a1-kube-api-access-bbd28\" (UniqueName: \"kubernetes.io/projected/aec931d7-85d3-48eb-9597-4e66cda196a1-kube-api-access-bbd28\") pod \"aec931d7-85d3-48eb-9597-4e66cda196a1\" (UID: \"aec931d7-85d3-48eb-9597-4e66cda196a1\") "
Oct 30 22:30:17.714824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.714722    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec931d7-85d3-48eb-9597-4e66cda196a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec931d7-85d3-48eb-9597-4e66cda196a1-maas-api-service-ca\") pod \"aec931d7-85d3-48eb-9597-4e66cda196a1\" (UID: \"aec931d7-85d3-48eb-9597-4e66cda196a1\") "
Oct 30 22:30:17.715146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.715115    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec931d7-85d3-48eb-9597-4e66cda196a1-maas-api-service-ca" pod "aec931d7-85d3-48eb-9597-4e66cda196a1" (UID: "aec931d7-85d3-48eb-9597-4e66cda196a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.715590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.715540    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec931d7-85d3-48eb-9597-4e66cda196a1-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:17.717137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.717104    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec931d7-85d3-48eb-9597-4e66cda196a1-kube-api-access-bbd28" pod "aec931d7-85d3-48eb-9597-4e66cda196a1" (UID: "aec931d7-85d3-48eb-9597-4e66cda196a1"). InnerVolumeSpecName "kube-api-access-bbd28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.817496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:17.817458    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbd28\" (UniqueName: \"kubernetes.io/projected/aec931d7-85d3-48eb-9597-4e66cda196a1-kube-api-access-bbd28\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:18.089539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.089504    4581 scope.go:122] "RemoveContainer" containerID="bc52102c9c295336c9230c70ca2f74cf5a36d3170ccbfb90c4cacda10a4d9132"
Oct 30 22:30:18.090041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.090012    4581 scope.go:122] "RemoveContainer" containerID="00daeb20ba7fd05e9dc032e033aaede7c21c0522107ad8b5286e475de424b8c5"
Oct 30 22:30:18.090314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.090295    4581 scope.go:122] "RemoveContainer" containerID="8abdab36cd39057ed2440ee3b1b4711e414c7f271a2a1bc40c517e4ec233341b"
Oct 30 22:30:18.369772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.369693    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" event={"ID":"2c8eb378-fa1e-4cce-9732-d67f5372f7e6","Type":"ContainerStarted","Data":"257035ef55af3b6a7f9b8f0bb968a9feaa8bf422512346253a85c9ed105519ac"}
Oct 30 22:30:18.378115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.378015    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" event={"ID":"5925fcc5-3727-4da2-a914-b916860e8323","Type":"ContainerStarted","Data":"0eba8e7262bd1767294455ea38acd25b0616cf2d578d3e3e550f50e305fde0a6"}
Oct 30 22:30:18.393701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.393650    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4" event={"ID":"aec931d7-85d3-48eb-9597-4e66cda196a1","Type":"ContainerDied","Data":"a551150d0dcd32be4a7655aaa5ea015b5ae63c217452f03482f1465a5b66df49"}
Oct 30 22:30:18.394676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.393724    4581 scope.go:122] "RemoveContainer" containerID="ff562dc0d0a5bbd6ed64c662d4ef6644b656b9d61820501547a173c70f061e19"
Oct 30 22:30:18.394676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.393811    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"
Oct 30 22:30:18.405332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.404707    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" event={"ID":"ab2f7876-eb7d-42a8-b95b-60fecb38488c","Type":"ContainerStarted","Data":"e1868a7c8098f34ad279c70cdcc895e73511ee3510af38948cfbca7018ffa7e0"}
Oct 30 22:30:18.406383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.406340    4581 scope.go:122] "RemoveContainer" containerID="e1868a7c8098f34ad279c70cdcc895e73511ee3510af38948cfbca7018ffa7e0"
Oct 30 22:30:18.407003 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:18.406975    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj_odh-ai-gateway-infra(ab2f7876-eb7d-42a8-b95b-60fecb38488c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c"
Oct 30 22:30:18.413493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.413458    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"]
Oct 30 22:30:18.477007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.476957    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"]
Oct 30 22:30:18.490509 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.490469    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"]
Oct 30 22:30:18.727265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.727173    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"]
Oct 30 22:30:18.727817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:18.727771    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy" containerID="cri-o://9f1c65faf8cb75b9e91aef8361bfc402fbba70affc3da950c2badb79b778668c" gracePeriod=30
Oct 30 22:30:19.033539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.033508    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"]
Oct 30 22:30:19.035166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.035140    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29889990-z8wc4"]
Oct 30 22:30:19.047063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.047033    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:19.147847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.147810    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7833a4f4-70d2-47aa-81df-6d42a71db4ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7833a4f4-70d2-47aa-81df-6d42a71db4ad-maas-api-service-ca\") pod \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\" (UID: \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\") "
Oct 30 22:30:19.148071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.148053    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7833a4f4-70d2-47aa-81df-6d42a71db4ad-kube-api-access-rmn87\" (UniqueName: \"kubernetes.io/projected/7833a4f4-70d2-47aa-81df-6d42a71db4ad-kube-api-access-rmn87\") pod \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\" (UID: \"7833a4f4-70d2-47aa-81df-6d42a71db4ad\") "
Oct 30 22:30:19.148941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.148908    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7833a4f4-70d2-47aa-81df-6d42a71db4ad-maas-api-service-ca" pod "7833a4f4-70d2-47aa-81df-6d42a71db4ad" (UID: "7833a4f4-70d2-47aa-81df-6d42a71db4ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.164548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.164509    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7833a4f4-70d2-47aa-81df-6d42a71db4ad-kube-api-access-rmn87" pod "7833a4f4-70d2-47aa-81df-6d42a71db4ad" (UID: "7833a4f4-70d2-47aa-81df-6d42a71db4ad"). InnerVolumeSpecName "kube-api-access-rmn87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.228898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.228872    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:19.233049 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.233027    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:19.237044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.237026    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:19.249987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.249930    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmn87\" (UniqueName: \"kubernetes.io/projected/7833a4f4-70d2-47aa-81df-6d42a71db4ad-kube-api-access-rmn87\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.249987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.249954    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7833a4f4-70d2-47aa-81df-6d42a71db4ad-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.351389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.351356    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b60772f-0227-4978-aecf-c38efbd91a15-kube-api-access-qhpsl\" (UniqueName: \"kubernetes.io/projected/9b60772f-0227-4978-aecf-c38efbd91a15-kube-api-access-qhpsl\") pod \"9b60772f-0227-4978-aecf-c38efbd91a15\" (UID: \"9b60772f-0227-4978-aecf-c38efbd91a15\") "
Oct 30 22:30:19.353522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.353494    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b60772f-0227-4978-aecf-c38efbd91a15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b60772f-0227-4978-aecf-c38efbd91a15-maas-api-service-ca\") pod \"9b60772f-0227-4978-aecf-c38efbd91a15\" (UID: \"9b60772f-0227-4978-aecf-c38efbd91a15\") "
Oct 30 22:30:19.353718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.353703    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-maas-api-service-ca\") pod \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\" (UID: \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\") "
Oct 30 22:30:19.354055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.354038    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-kube-api-access-plxrm\" (UniqueName: \"kubernetes.io/projected/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-kube-api-access-plxrm\") pod \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\" (UID: \"aec3f64c-7c9b-4f83-8b0d-07dd337224b6\") "
Oct 30 22:30:19.354866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.354841    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9bc499-e4f5-42ba-ad63-9a98580a7838-kube-api-access-n78st\" (UniqueName: \"kubernetes.io/projected/2b9bc499-e4f5-42ba-ad63-9a98580a7838-kube-api-access-n78st\") pod \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\" (UID: \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\") "
Oct 30 22:30:19.355332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.355119    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-maas-api-service-ca" pod "aec3f64c-7c9b-4f83-8b0d-07dd337224b6" (UID: "aec3f64c-7c9b-4f83-8b0d-07dd337224b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.355659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.355433    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b60772f-0227-4978-aecf-c38efbd91a15-maas-api-service-ca" pod "9b60772f-0227-4978-aecf-c38efbd91a15" (UID: "9b60772f-0227-4978-aecf-c38efbd91a15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.355753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.355364    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9bc499-e4f5-42ba-ad63-9a98580a7838-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9bc499-e4f5-42ba-ad63-9a98580a7838-maas-api-service-ca\") pod \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\" (UID: \"2b9bc499-e4f5-42ba-ad63-9a98580a7838\") "
Oct 30 22:30:19.356148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.356091    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9bc499-e4f5-42ba-ad63-9a98580a7838-maas-api-service-ca" pod "2b9bc499-e4f5-42ba-ad63-9a98580a7838" (UID: "2b9bc499-e4f5-42ba-ad63-9a98580a7838"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.356817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.356791    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9bc499-e4f5-42ba-ad63-9a98580a7838-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.356945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.356932    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b60772f-0227-4978-aecf-c38efbd91a15-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.357028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.357017    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.365569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.364933    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b60772f-0227-4978-aecf-c38efbd91a15-kube-api-access-qhpsl" pod "9b60772f-0227-4978-aecf-c38efbd91a15" (UID: "9b60772f-0227-4978-aecf-c38efbd91a15"). InnerVolumeSpecName "kube-api-access-qhpsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.366798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.366767    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9bc499-e4f5-42ba-ad63-9a98580a7838-kube-api-access-n78st" pod "2b9bc499-e4f5-42ba-ad63-9a98580a7838" (UID: "2b9bc499-e4f5-42ba-ad63-9a98580a7838"). InnerVolumeSpecName "kube-api-access-n78st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.367645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.367623    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-kube-api-access-plxrm" pod "aec3f64c-7c9b-4f83-8b0d-07dd337224b6" (UID: "aec3f64c-7c9b-4f83-8b0d-07dd337224b6"). InnerVolumeSpecName "kube-api-access-plxrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.420678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.420523    4581 generic.go:356] "Generic (PLEG): container finished" podID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerID="013f46788cffbc2e90f2b75eb927264e03dfd357fd3a5948accd890c2b54a639" exitCode=60
Oct 30 22:30:19.420678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.420589    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" event={"ID":"02cae3ab-37b3-4947-9e3c-b37bb36318e9","Type":"ContainerDied","Data":"013f46788cffbc2e90f2b75eb927264e03dfd357fd3a5948accd890c2b54a639"}
Oct 30 22:30:19.420678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.420656    4581 scope.go:122] "RemoveContainer" containerID="8abdab36cd39057ed2440ee3b1b4711e414c7f271a2a1bc40c517e4ec233341b"
Oct 30 22:30:19.421168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.421106    4581 scope.go:122] "RemoveContainer" containerID="013f46788cffbc2e90f2b75eb927264e03dfd357fd3a5948accd890c2b54a639"
Oct 30 22:30:19.421645 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:19.421617    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l_odh-ai-gateway-infra(02cae3ab-37b3-4947-9e3c-b37bb36318e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9"
Oct 30 22:30:19.422963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.422945    4581 generic.go:356] "Generic (PLEG): container finished" podID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerID="257035ef55af3b6a7f9b8f0bb968a9feaa8bf422512346253a85c9ed105519ac" exitCode=60
Oct 30 22:30:19.423055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.423001    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" event={"ID":"2c8eb378-fa1e-4cce-9732-d67f5372f7e6","Type":"ContainerDied","Data":"257035ef55af3b6a7f9b8f0bb968a9feaa8bf422512346253a85c9ed105519ac"}
Oct 30 22:30:19.423545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.423528    4581 scope.go:122] "RemoveContainer" containerID="257035ef55af3b6a7f9b8f0bb968a9feaa8bf422512346253a85c9ed105519ac"
Oct 30 22:30:19.423923 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:19.423904    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr_odh-ai-gateway-infra(2c8eb378-fa1e-4cce-9732-d67f5372f7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6"
Oct 30 22:30:19.425041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.425021    4581 generic.go:356] "Generic (PLEG): container finished" podID="5925fcc5-3727-4da2-a914-b916860e8323" containerID="0eba8e7262bd1767294455ea38acd25b0616cf2d578d3e3e550f50e305fde0a6" exitCode=60
Oct 30 22:30:19.425129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.425093    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" event={"ID":"5925fcc5-3727-4da2-a914-b916860e8323","Type":"ContainerDied","Data":"0eba8e7262bd1767294455ea38acd25b0616cf2d578d3e3e550f50e305fde0a6"}
Oct 30 22:30:19.425466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.425451    4581 scope.go:122] "RemoveContainer" containerID="0eba8e7262bd1767294455ea38acd25b0616cf2d578d3e3e550f50e305fde0a6"
Oct 30 22:30:19.425789 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:19.425771    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l_odh-ai-gateway-infra(5925fcc5-3727-4da2-a914-b916860e8323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" podUID="5925fcc5-3727-4da2-a914-b916860e8323"
Oct 30 22:30:19.427412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.427392    4581 generic.go:356] "Generic (PLEG): container finished" podID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerID="b1e7b73c01ecdf88630e070e589e661ecea857e3a8ed67693220d6abb92bce65" exitCode=60
Oct 30 22:30:19.427508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.427464    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" event={"ID":"811ad8ba-e719-4481-a4ae-a13bf468de62","Type":"ContainerDied","Data":"b1e7b73c01ecdf88630e070e589e661ecea857e3a8ed67693220d6abb92bce65"}
Oct 30 22:30:19.427802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.427784    4581 scope.go:122] "RemoveContainer" containerID="b1e7b73c01ecdf88630e070e589e661ecea857e3a8ed67693220d6abb92bce65"
Oct 30 22:30:19.428088 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:19.428064    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9_odh-ai-gateway-infra(811ad8ba-e719-4481-a4ae-a13bf468de62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62"
Oct 30 22:30:19.428821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.428803    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"
Oct 30 22:30:19.428908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.428824    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5" event={"ID":"7833a4f4-70d2-47aa-81df-6d42a71db4ad","Type":"ContainerDied","Data":"4c9d626b5792503e59d43b3d35f998f1f6eb0404ba8531186e291c7b3533a018"}
Oct 30 22:30:19.430137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.430120    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"
Oct 30 22:30:19.430256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.430158    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj" event={"ID":"9b60772f-0227-4978-aecf-c38efbd91a15","Type":"ContainerDied","Data":"78960efb68754f2d29401fecf2079667421646d2dfd706f336e02b0989a403e6"}
Oct 30 22:30:19.431659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.431638    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"
Oct 30 22:30:19.431743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.431662    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5" event={"ID":"aec3f64c-7c9b-4f83-8b0d-07dd337224b6","Type":"ContainerDied","Data":"5407bf33dbfea670d8626e53cbad12316a484b0535e23044b525a433ca454bce"}
Oct 30 22:30:19.435966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.435942    4581 generic.go:356] "Generic (PLEG): container finished" podID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerID="d022f1406028cbad3ae9102b8b0a897b79c1a931e0f7b56cfeec3276b2388b2c" exitCode=60
Oct 30 22:30:19.435966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.435954    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" event={"ID":"a3ce3beb-2566-4e01-92b0-02ebc5048a5d","Type":"ContainerDied","Data":"d022f1406028cbad3ae9102b8b0a897b79c1a931e0f7b56cfeec3276b2388b2c"}
Oct 30 22:30:19.436329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.436312    4581 scope.go:122] "RemoveContainer" containerID="d022f1406028cbad3ae9102b8b0a897b79c1a931e0f7b56cfeec3276b2388b2c"
Oct 30 22:30:19.436612 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:19.436592    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp_odh-ai-gateway-infra(a3ce3beb-2566-4e01-92b0-02ebc5048a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d"
Oct 30 22:30:19.438075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.438055    4581 generic.go:356] "Generic (PLEG): container finished" podID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerID="e1868a7c8098f34ad279c70cdcc895e73511ee3510af38948cfbca7018ffa7e0" exitCode=60
Oct 30 22:30:19.438152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.438120    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" event={"ID":"ab2f7876-eb7d-42a8-b95b-60fecb38488c","Type":"ContainerDied","Data":"e1868a7c8098f34ad279c70cdcc895e73511ee3510af38948cfbca7018ffa7e0"}
Oct 30 22:30:19.438543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.438525    4581 scope.go:122] "RemoveContainer" containerID="e1868a7c8098f34ad279c70cdcc895e73511ee3510af38948cfbca7018ffa7e0"
Oct 30 22:30:19.438771 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:19.438750    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj_odh-ai-gateway-infra(ab2f7876-eb7d-42a8-b95b-60fecb38488c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c"
Oct 30 22:30:19.439621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.439601    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt" event={"ID":"2b9bc499-e4f5-42ba-ad63-9a98580a7838","Type":"ContainerDied","Data":"381993cc9624fd8dae477dda1d1b08aacb5f89f0dffce4a24fa570afc5e40bb6"}
Oct 30 22:30:19.439738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.439611    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"
Oct 30 22:30:19.445954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.445931    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"]
Oct 30 22:30:19.458423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.458395    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78st\" (UniqueName: \"kubernetes.io/projected/2b9bc499-e4f5-42ba-ad63-9a98580a7838-kube-api-access-n78st\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.458549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.458434    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhpsl\" (UniqueName: \"kubernetes.io/projected/9b60772f-0227-4978-aecf-c38efbd91a15-kube-api-access-qhpsl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.458549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.458451    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plxrm\" (UniqueName: \"kubernetes.io/projected/aec3f64c-7c9b-4f83-8b0d-07dd337224b6-kube-api-access-plxrm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.482724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.482705    4581 scope.go:122] "RemoveContainer" containerID="0000886d2b398de8c677337254f20fc28549c60edfc49a4237a7616ad50870da"
Oct 30 22:30:19.505508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.505480    4581 scope.go:122] "RemoveContainer" containerID="7f754eea07e89b9f1f9960792da18b1e490f7d084e4fcef74e784cc81c686d27"
Oct 30 22:30:19.521419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.521394    4581 scope.go:122] "RemoveContainer" containerID="00daeb20ba7fd05e9dc032e033aaede7c21c0522107ad8b5286e475de424b8c5"
Oct 30 22:30:19.526728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.526700    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"]
Oct 30 22:30:19.540324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.540298    4581 scope.go:122] "RemoveContainer" containerID="8901c3181b5946388dc584366ad2aa6fceb070beb986b91c756d97c668e51637"
Oct 30 22:30:19.555926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.555897    4581 scope.go:122] "RemoveContainer" containerID="cac92f0040075bb6bbb4d89b60a83a6530703ed3a6c5e4447c142dbabe8ae633"
Oct 30 22:30:19.571431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.571180    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"]
Oct 30 22:30:19.577634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.577376    4581 scope.go:122] "RemoveContainer" containerID="fde35add84181dd31af5bf95df7f72472d409232c793af1df136c3a357721a50"
Oct 30 22:30:19.602188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.602161    4581 scope.go:122] "RemoveContainer" containerID="bc52102c9c295336c9230c70ca2f74cf5a36d3170ccbfb90c4cacda10a4d9132"
Oct 30 22:30:19.625207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.625182    4581 scope.go:122] "RemoveContainer" containerID="b096e5d53d9de95b2f16fe0aa3fff6b4c357c32a0bed5d5223fa8b9b1e3cda92"
Oct 30 22:30:19.641641 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.641620    4581 scope.go:122] "RemoveContainer" containerID="f2f9d3bd4a591ec50eee64196011fe7853595fa261bfa5af1683bcbbc3276964"
Oct 30 22:30:19.730021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.729987    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"]
Oct 30 22:30:19.736720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.736687    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29889990-xdsr5"]
Oct 30 22:30:19.772428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.772333    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"]
Oct 30 22:30:19.779096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.779064    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29889990-wn6vt"]
Oct 30 22:30:19.821781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.821723    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"]
Oct 30 22:30:19.827674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.827640    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d52c59-29889990-cz8v5"]
Oct 30 22:30:19.859296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.859261    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"]
Oct 30 22:30:19.862620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.862593    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"]
Oct 30 22:30:19.864040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.864014    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:19.864271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.864207    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:19.865089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865051    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:19.865239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865205    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:19.865336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865324    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:19.865428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865418    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:19.865511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865502    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:19.865610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865599    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:19.865697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865688    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:19.865774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865765    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:19.865854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865846    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:19.865923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.865912    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:19.866012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866004    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:19.866081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866072    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:19.866164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866156    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:19.866255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866244    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:19.866331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866322    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:19.866406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866397    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:19.866528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866519    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa" containerName="cleanup"
Oct 30 22:30:19.866597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866588    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa" containerName="cleanup"
Oct 30 22:30:19.866748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866676    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa" containerName="cleanup"
Oct 30 22:30:19.866835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866825    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa" containerName="cleanup"
Oct 30 22:30:19.866930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866921    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:19.867005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.866995    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:19.867072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867063    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:19.867152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867143    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:19.867443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867429    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:19.867541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867530    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa" containerName="cleanup"
Oct 30 22:30:19.867615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867607    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:19.867685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867677    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc6bbad-d0d8-4154-ae36-416ace169faa" containerName="cleanup"
Oct 30 22:30:19.867752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867745    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:19.867896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867886    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:19.867967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.867959    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:19.868039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.868031    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:19.868110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.868102    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:19.868188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.868178    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:19.868286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.868274    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:19.868377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.868368    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:19.868464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.868455    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:19.874957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.874935    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29889990-p2mvj"]
Oct 30 22:30:19.875113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.875027    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.891612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.891453    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-30edd7-openshift-default-dockercfg-87hn4\""
Oct 30 22:30:19.897984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.897946    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"]
Oct 30 22:30:19.970203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970161    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-certs\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.970427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970308    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsfq\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-kube-api-access-zfsfq\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.970427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970370    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-envoy\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.970539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970512    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-credential-socket\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.970605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970588    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istiod-ca-cert\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.970658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970638    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-token\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.970779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970751    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-data\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.970928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970813    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-socket\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:19.970928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:19.970882    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-podinfo\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072432    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-certs\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072532    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsfq\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-kube-api-access-zfsfq\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072577    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-envoy\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072628    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-credential-socket\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072775    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istiod-ca-cert\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072844    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-token\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072896    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-data\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072910    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-certs\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072943    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-socket\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.072995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.072980    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-podinfo\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.073238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.073023    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-credential-socket\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.073374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.073354    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-socket\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.073437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.073380    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-data\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.073592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.073563    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istiod-ca-cert\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.075748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.075719    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-envoy\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.075889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.075867    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-podinfo\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.091672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.091638    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-token\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.093468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.093434    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsfq\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-kube-api-access-zfsfq\") pod \"e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") " pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.135243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.128016    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" path="/var/lib/kubelet/pods/2b9bc499-e4f5-42ba-ad63-9a98580a7838/volumes"
Oct 30 22:30:20.135243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.128836    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" path="/var/lib/kubelet/pods/7833a4f4-70d2-47aa-81df-6d42a71db4ad/volumes"
Oct 30 22:30:20.135243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.129478    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" path="/var/lib/kubelet/pods/9b60772f-0227-4978-aecf-c38efbd91a15/volumes"
Oct 30 22:30:20.155309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.141918    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" path="/var/lib/kubelet/pods/aec3f64c-7c9b-4f83-8b0d-07dd337224b6/volumes"
Oct 30 22:30:20.155309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.142643    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" path="/var/lib/kubelet/pods/aec931d7-85d3-48eb-9597-4e66cda196a1/volumes"
Oct 30 22:30:20.196601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.196547    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:20.481085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.481048    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"]
Oct 30 22:30:20.665071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.665024    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"]
Oct 30 22:30:20.677062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.676992    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"]
Oct 30 22:30:20.678865 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:20.678835    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6511e97e_31ed_4652_b7a1_e0661ea7c88e.slice/crio-609d23e7952a1483098b11d786d2f098ee3f2e981553c5a0509dcb9f13b338f1 WatchSource:0}: Error finding container 609d23e7952a1483098b11d786d2f098ee3f2e981553c5a0509dcb9f13b338f1: Status 404 returned error can't find the container with id 609d23e7952a1483098b11d786d2f098ee3f2e981553c5a0509dcb9f13b338f1
Oct 30 22:30:20.702988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.702046    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:20.702988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.702148    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:20.702988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.702192    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:20.725480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.724116    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"]
Oct 30 22:30:20.812323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.812292    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:20.838837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.838744    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:20.870813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.870782    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-maas-api-service-ca\") pod \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\" (UID: \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\") "
Oct 30 22:30:20.871450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.871430    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5925fcc5-3727-4da2-a914-b916860e8323-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925fcc5-3727-4da2-a914-b916860e8323-maas-api-service-ca\") pod \"5925fcc5-3727-4da2-a914-b916860e8323\" (UID: \"5925fcc5-3727-4da2-a914-b916860e8323\") "
Oct 30 22:30:20.872278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.872253    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-kube-api-access-fnphr\" (UniqueName: \"kubernetes.io/projected/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-kube-api-access-fnphr\") pod \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\" (UID: \"2c8eb378-fa1e-4cce-9732-d67f5372f7e6\") "
Oct 30 22:30:20.873068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.873047    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5925fcc5-3727-4da2-a914-b916860e8323-kube-api-access-42cxh\" (UniqueName: \"kubernetes.io/projected/5925fcc5-3727-4da2-a914-b916860e8323-kube-api-access-42cxh\") pod \"5925fcc5-3727-4da2-a914-b916860e8323\" (UID: \"5925fcc5-3727-4da2-a914-b916860e8323\") "
Oct 30 22:30:20.874014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.871369    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-maas-api-service-ca" pod "2c8eb378-fa1e-4cce-9732-d67f5372f7e6" (UID: "2c8eb378-fa1e-4cce-9732-d67f5372f7e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.874138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.871920    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5925fcc5-3727-4da2-a914-b916860e8323-maas-api-service-ca" pod "5925fcc5-3727-4da2-a914-b916860e8323" (UID: "5925fcc5-3727-4da2-a914-b916860e8323"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.875120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.875100    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:20.875367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.875354    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925fcc5-3727-4da2-a914-b916860e8323-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:20.882120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.882072    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5925fcc5-3727-4da2-a914-b916860e8323-kube-api-access-42cxh" pod "5925fcc5-3727-4da2-a914-b916860e8323" (UID: "5925fcc5-3727-4da2-a914-b916860e8323"). InnerVolumeSpecName "kube-api-access-42cxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.882477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.882442    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-kube-api-access-fnphr" pod "2c8eb378-fa1e-4cce-9732-d67f5372f7e6" (UID: "2c8eb378-fa1e-4cce-9732-d67f5372f7e6"). InnerVolumeSpecName "kube-api-access-fnphr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.906094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.902380    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:20.916283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.916124    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:20.989230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.988322    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02cae3ab-37b3-4947-9e3c-b37bb36318e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cae3ab-37b3-4947-9e3c-b37bb36318e9-maas-api-service-ca\") pod \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\" (UID: \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\") "
Oct 30 22:30:20.990655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.989440    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2f7876-eb7d-42a8-b95b-60fecb38488c-kube-api-access-9ws68\" (UniqueName: \"kubernetes.io/projected/ab2f7876-eb7d-42a8-b95b-60fecb38488c-kube-api-access-9ws68\") pod \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\" (UID: \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\") "
Oct 30 22:30:20.990655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.989593    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cae3ab-37b3-4947-9e3c-b37bb36318e9-kube-api-access-j9fkp\" (UniqueName: \"kubernetes.io/projected/02cae3ab-37b3-4947-9e3c-b37bb36318e9-kube-api-access-j9fkp\") pod \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\" (UID: \"02cae3ab-37b3-4947-9e3c-b37bb36318e9\") "
Oct 30 22:30:20.990655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.989646    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2f7876-eb7d-42a8-b95b-60fecb38488c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f7876-eb7d-42a8-b95b-60fecb38488c-maas-api-service-ca\") pod \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\" (UID: \"ab2f7876-eb7d-42a8-b95b-60fecb38488c\") "
Oct 30 22:30:20.990655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.990592    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnphr\" (UniqueName: \"kubernetes.io/projected/2c8eb378-fa1e-4cce-9732-d67f5372f7e6-kube-api-access-fnphr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:20.990655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.990617    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42cxh\" (UniqueName: \"kubernetes.io/projected/5925fcc5-3727-4da2-a914-b916860e8323-kube-api-access-42cxh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:20.991645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.991607    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2f7876-eb7d-42a8-b95b-60fecb38488c-maas-api-service-ca" pod "ab2f7876-eb7d-42a8-b95b-60fecb38488c" (UID: "ab2f7876-eb7d-42a8-b95b-60fecb38488c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.992494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.992468    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02cae3ab-37b3-4947-9e3c-b37bb36318e9-maas-api-service-ca" pod "02cae3ab-37b3-4947-9e3c-b37bb36318e9" (UID: "02cae3ab-37b3-4947-9e3c-b37bb36318e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.997284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:20.997244    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2f7876-eb7d-42a8-b95b-60fecb38488c-kube-api-access-9ws68" pod "ab2f7876-eb7d-42a8-b95b-60fecb38488c" (UID: "ab2f7876-eb7d-42a8-b95b-60fecb38488c"). InnerVolumeSpecName "kube-api-access-9ws68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:21.028270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.027574    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cae3ab-37b3-4947-9e3c-b37bb36318e9-kube-api-access-j9fkp" pod "02cae3ab-37b3-4947-9e3c-b37bb36318e9" (UID: "02cae3ab-37b3-4947-9e3c-b37bb36318e9"). InnerVolumeSpecName "kube-api-access-j9fkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:21.045629 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.045595    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:21.094897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.094850    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-kube-api-access-ntbd4\" (UniqueName: \"kubernetes.io/projected/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-kube-api-access-ntbd4\") pod \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\" (UID: \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\") "
Oct 30 22:30:21.095290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.095264    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-maas-api-service-ca\") pod \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\" (UID: \"a3ce3beb-2566-4e01-92b0-02ebc5048a5d\") "
Oct 30 22:30:21.096364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.096340    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ws68\" (UniqueName: \"kubernetes.io/projected/ab2f7876-eb7d-42a8-b95b-60fecb38488c-kube-api-access-9ws68\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:21.096470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.096368    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fkp\" (UniqueName: \"kubernetes.io/projected/02cae3ab-37b3-4947-9e3c-b37bb36318e9-kube-api-access-j9fkp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:21.096470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.096389    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f7876-eb7d-42a8-b95b-60fecb38488c-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:21.096470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.096404    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02cae3ab-37b3-4947-9e3c-b37bb36318e9-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:21.096946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.096919    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-maas-api-service-ca" pod "a3ce3beb-2566-4e01-92b0-02ebc5048a5d" (UID: "a3ce3beb-2566-4e01-92b0-02ebc5048a5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:21.099849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.099822    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-kube-api-access-ntbd4" pod "a3ce3beb-2566-4e01-92b0-02ebc5048a5d" (UID: "a3ce3beb-2566-4e01-92b0-02ebc5048a5d"). InnerVolumeSpecName "kube-api-access-ntbd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:21.127917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.127888    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:21.201251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.199823    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811ad8ba-e719-4481-a4ae-a13bf468de62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/811ad8ba-e719-4481-a4ae-a13bf468de62-maas-api-service-ca\") pod \"811ad8ba-e719-4481-a4ae-a13bf468de62\" (UID: \"811ad8ba-e719-4481-a4ae-a13bf468de62\") "
Oct 30 22:30:21.201251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.200114    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811ad8ba-e719-4481-a4ae-a13bf468de62-kube-api-access-59nn5\" (UniqueName: \"kubernetes.io/projected/811ad8ba-e719-4481-a4ae-a13bf468de62-kube-api-access-59nn5\") pod \"811ad8ba-e719-4481-a4ae-a13bf468de62\" (UID: \"811ad8ba-e719-4481-a4ae-a13bf468de62\") "
Oct 30 22:30:21.201251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.200446    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811ad8ba-e719-4481-a4ae-a13bf468de62-maas-api-service-ca" pod "811ad8ba-e719-4481-a4ae-a13bf468de62" (UID: "811ad8ba-e719-4481-a4ae-a13bf468de62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:21.201251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.201072    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntbd4\" (UniqueName: \"kubernetes.io/projected/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-kube-api-access-ntbd4\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:21.201251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.201095    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ce3beb-2566-4e01-92b0-02ebc5048a5d-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:21.201251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.201109    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/811ad8ba-e719-4481-a4ae-a13bf468de62-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:21.204250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.203412    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811ad8ba-e719-4481-a4ae-a13bf468de62-kube-api-access-59nn5" pod "811ad8ba-e719-4481-a4ae-a13bf468de62" (UID: "811ad8ba-e719-4481-a4ae-a13bf468de62"). InnerVolumeSpecName "kube-api-access-59nn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:21.303262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.303189    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59nn5\" (UniqueName: \"kubernetes.io/projected/811ad8ba-e719-4481-a4ae-a13bf468de62-kube-api-access-59nn5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:21.486723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.486624    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr" event={"ID":"2c8eb378-fa1e-4cce-9732-d67f5372f7e6","Type":"ContainerDied","Data":"8b6ac6f9b145ea5645de3b92ff96152110f11f3c456d5521593f04730bac2a38"}
Oct 30 22:30:21.486723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.486642    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"
Oct 30 22:30:21.486723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.486694    4581 scope.go:122] "RemoveContainer" containerID="257035ef55af3b6a7f9b8f0bb968a9feaa8bf422512346253a85c9ed105519ac"
Oct 30 22:30:21.488761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.488727    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l" event={"ID":"5925fcc5-3727-4da2-a914-b916860e8323","Type":"ContainerDied","Data":"6cf3ddf4b128751b20356be5454ee9e90b7a660a76c305585f555769811a2ca3"}
Oct 30 22:30:21.488973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.488742    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"
Oct 30 22:30:21.490932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.490905    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9" event={"ID":"811ad8ba-e719-4481-a4ae-a13bf468de62","Type":"ContainerDied","Data":"97e3ed52983b0e974ddae11984a754005fcd7ff776ce0543fb6e1b3d7fab8d20"}
Oct 30 22:30:21.492340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.490912    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"
Oct 30 22:30:21.494888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.494858    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" event={"ID":"6511e97e-31ed-4652-b7a1-e0661ea7c88e","Type":"ContainerStarted","Data":"9d9f02ca54d4c40c8b19ef4d06279b7cc8dd9b53c890b17c0f3aa19c6d09cb55"}
Oct 30 22:30:21.494993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.494913    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" event={"ID":"6511e97e-31ed-4652-b7a1-e0661ea7c88e","Type":"ContainerStarted","Data":"609d23e7952a1483098b11d786d2f098ee3f2e981553c5a0509dcb9f13b338f1"}
Oct 30 22:30:21.497025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.497003    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"
Oct 30 22:30:21.498092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.498071    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp" event={"ID":"a3ce3beb-2566-4e01-92b0-02ebc5048a5d","Type":"ContainerDied","Data":"80655847a9793893788ad3a95073a3d97036d2331a1fc2e585e00d4585e542f3"}
Oct 30 22:30:21.498981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.498960    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj" event={"ID":"ab2f7876-eb7d-42a8-b95b-60fecb38488c","Type":"ContainerDied","Data":"e2a08cc4d2799188fea3cc8244247754880faf2e498ff5a37d4706e43dd3e1fc"}
Oct 30 22:30:21.499082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.499055    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"
Oct 30 22:30:21.501136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.501114    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l" event={"ID":"02cae3ab-37b3-4947-9e3c-b37bb36318e9","Type":"ContainerDied","Data":"165b2e0936590521b6ecba678f4a60476109067505eb22fea30f07406e2b4e26"}
Oct 30 22:30:21.501258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.501159    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"
Oct 30 22:30:21.502527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.502496    4581 scope.go:122] "RemoveContainer" containerID="0eba8e7262bd1767294455ea38acd25b0616cf2d578d3e3e550f50e305fde0a6"
Oct 30 22:30:21.521589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.521568    4581 scope.go:122] "RemoveContainer" containerID="b1e7b73c01ecdf88630e070e589e661ecea857e3a8ed67693220d6abb92bce65"
Oct 30 22:30:21.535848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.535828    4581 scope.go:122] "RemoveContainer" containerID="d022f1406028cbad3ae9102b8b0a897b79c1a931e0f7b56cfeec3276b2388b2c"
Oct 30 22:30:21.544008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.543952    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podStartSLOduration=2.543934321 podStartE2EDuration="2.543934321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:21.538586569 +0000 UTC m=+1405.577525455" watchObservedRunningTime="2026-10-30 22:30:21.543934321 +0000 UTC m=+1405.582873210"
Oct 30 22:30:21.549916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.549887    4581 scope.go:122] "RemoveContainer" containerID="e1868a7c8098f34ad279c70cdcc895e73511ee3510af38948cfbca7018ffa7e0"
Oct 30 22:30:21.570341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.570322    4581 scope.go:122] "RemoveContainer" containerID="013f46788cffbc2e90f2b75eb927264e03dfd357fd3a5948accd890c2b54a639"
Oct 30 22:30:21.640681 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.640649    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"]
Oct 30 22:30:21.646151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.646121    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29889990-lkqzr"]
Oct 30 22:30:21.709241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.703574    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"]
Oct 30 22:30:21.720064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.719999    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29889990-gbnzp"]
Oct 30 22:30:21.766298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.765042    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"]
Oct 30 22:30:21.766298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.765094    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29889990-r7ld9"]
Oct 30 22:30:21.773038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.773008    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"]
Oct 30 22:30:21.778522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.778498    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29889990-28n2l"]
Oct 30 22:30:21.804923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.804890    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"]
Oct 30 22:30:21.809795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.809764    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29889990-bgs7l"]
Oct 30 22:30:21.836180 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.836148    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"]
Oct 30 22:30:21.843966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:21.843935    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29889990-9k9qj"]
Oct 30 22:30:22.100873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.100788    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" path="/var/lib/kubelet/pods/02cae3ab-37b3-4947-9e3c-b37bb36318e9/volumes"
Oct 30 22:30:22.101475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.101447    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" path="/var/lib/kubelet/pods/2c8eb378-fa1e-4cce-9732-d67f5372f7e6/volumes"
Oct 30 22:30:22.101946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.101926    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5925fcc5-3727-4da2-a914-b916860e8323" path="/var/lib/kubelet/pods/5925fcc5-3727-4da2-a914-b916860e8323/volumes"
Oct 30 22:30:22.102719 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.102697    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" path="/var/lib/kubelet/pods/811ad8ba-e719-4481-a4ae-a13bf468de62/volumes"
Oct 30 22:30:22.103169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.103149    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" path="/var/lib/kubelet/pods/a3ce3beb-2566-4e01-92b0-02ebc5048a5d/volumes"
Oct 30 22:30:22.103670 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.103652    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" path="/var/lib/kubelet/pods/ab2f7876-eb7d-42a8-b95b-60fecb38488c/volumes"
Oct 30 22:30:22.197531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.197495    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:22.204139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.204109    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:22.517748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.517722    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:22.518642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:22.518625    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:30:24.602305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.594236    4581 generic.go:356] "Generic (PLEG): container finished" podID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerID="9f1c65faf8cb75b9e91aef8361bfc402fbba70affc3da950c2badb79b778668c" exitCode=0
Oct 30 22:30:24.602305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.595311    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" event={"ID":"043d11d3-326a-4c67-817a-7b9a7cdf9596","Type":"ContainerDied","Data":"9f1c65faf8cb75b9e91aef8361bfc402fbba70affc3da950c2badb79b778668c"}
Oct 30 22:30:24.602305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.595377    4581 scope.go:122] "RemoveContainer" containerID="e7480b2d393ec706c625bb4f3e72c7b51aef309dd99879cce3531c91ecd09408"
Oct 30 22:30:24.713436 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.713407    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:30:24.807375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807333    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043d11d3-326a-4c67-817a-7b9a7cdf9596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d11d3-326a-4c67-817a-7b9a7cdf9596-istiod-ca-cert\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807387    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-data\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807442    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-kube-api-access-gk72x\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-kube-api-access-gk72x\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807492    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-certs\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807583    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-socket\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807651    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-credential-socket\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807687    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-token\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-token\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807732    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-envoy\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807764    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-podinfo\") pod \"043d11d3-326a-4c67-817a-7b9a7cdf9596\" (UID: \"043d11d3-326a-4c67-817a-7b9a7cdf9596\") "
Oct 30 22:30:24.807950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807793    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-data" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.807950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.807865    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-certs" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.808103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.808071    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-socket" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.808103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.808085    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-credential-socket" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.808208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.808130    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043d11d3-326a-4c67-817a-7b9a7cdf9596-istiod-ca-cert" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:24.808735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.808712    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d11d3-326a-4c67-817a-7b9a7cdf9596-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.808735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.808739    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.808946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.808752    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.808946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.808764    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.808946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.808777    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.810772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.810735    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-podinfo" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:24.810972 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.810944    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-envoy" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.811074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.811006    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-token" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.811074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.811042    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-kube-api-access-gk72x" pod "043d11d3-326a-4c67-817a-7b9a7cdf9596" (UID: "043d11d3-326a-4c67-817a-7b9a7cdf9596"). InnerVolumeSpecName "kube-api-access-gk72x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.910483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.910453    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.910483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.910479    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.910483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.910488    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d11d3-326a-4c67-817a-7b9a7cdf9596-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.910753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:24.910499    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk72x\" (UniqueName: \"kubernetes.io/projected/043d11d3-326a-4c67-817a-7b9a7cdf9596-kube-api-access-gk72x\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:30:25.608323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:25.608282    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf" event={"ID":"043d11d3-326a-4c67-817a-7b9a7cdf9596","Type":"ContainerDied","Data":"f38f31828e9b3c77ec0f63d75d5b92265770fcc495eb1479ed14666534cd5d9e"}
Oct 30 22:30:25.608800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:25.608375    4581 scope.go:122] "RemoveContainer" containerID="9f1c65faf8cb75b9e91aef8361bfc402fbba70affc3da950c2badb79b778668c"
Oct 30 22:30:25.608800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:25.608302    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"
Oct 30 22:30:25.669429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:25.669391    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"]
Oct 30 22:30:25.676731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:25.676697    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-4z9xf"]
Oct 30 22:30:26.012554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.012464    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"]
Oct 30 22:30:26.013411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013386    4581 cpu_manager.go:414] "containerMap: removing container" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy"
Oct 30 22:30:26.013411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013409    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013419    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013426    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013450    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013457    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013471    4581 cpu_manager.go:414] "containerMap: removing container" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013478    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013502    4581 cpu_manager.go:414] "containerMap: removing container" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013510    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013539    4581 cpu_manager.go:414] "containerMap: removing container" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013546    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013555    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013562    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013572    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013579    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013594    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013601    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013614    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:26.013625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013621    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013639    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013646    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b60772f-0227-4978-aecf-c38efbd91a15" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013666    4581 cpu_manager.go:414] "containerMap: removing container" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013673    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013687    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013693    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013730    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013738    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013751    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013759    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013771    4581 cpu_manager.go:414] "containerMap: removing container" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013778    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013799    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013807    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013820    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013829    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013878    4581 cpu_manager.go:414] "containerMap: removing container" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013887    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013910    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.013917    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014088    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014101    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014110    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec3f64c-7c9b-4f83-8b0d-07dd337224b6" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014119    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014127    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014138    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014147    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014159    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014169    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014178    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014187    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014201    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" containerName="istio-proxy"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014210    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014242    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014251    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9bc499-e4f5-42ba-ad63-9a98580a7838" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014262    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014271    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec931d7-85d3-48eb-9597-4e66cda196a1" containerName="cleanup"
Oct 30 22:30:26.014480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.014282    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7833a4f4-70d2-47aa-81df-6d42a71db4ad" containerName="cleanup"
Oct 30 22:30:26.020366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.020343    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:26.033357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.033331    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-30edd7-dockercfg-jr9tj\""
Oct 30 22:30:26.033468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.033438    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-30edd7\""
Oct 30 22:30:26.037119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.037091    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"]
Oct 30 22:30:26.086114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.086077    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"]
Oct 30 22:30:26.087265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087242    4581 cpu_manager.go:414] "containerMap: removing container" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.087370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087267    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.087370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087291    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.087370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087299    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.087370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087323    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.087370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087331    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.087370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087365    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.087565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087374    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.087624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087610    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="811ad8ba-e719-4481-a4ae-a13bf468de62" containerName="cleanup"
Oct 30 22:30:26.087657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087635    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cae3ab-37b3-4947-9e3c-b37bb36318e9" containerName="cleanup"
Oct 30 22:30:26.087657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087648    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5925fcc5-3727-4da2-a914-b916860e8323" containerName="cleanup"
Oct 30 22:30:26.087729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087662    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ce3beb-2566-4e01-92b0-02ebc5048a5d" containerName="cleanup"
Oct 30 22:30:26.087729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087676    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2f7876-eb7d-42a8-b95b-60fecb38488c" containerName="cleanup"
Oct 30 22:30:26.087729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.087686    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8eb378-fa1e-4cce-9732-d67f5372f7e6" containerName="cleanup"
Oct 30 22:30:26.096342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.096311    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:26.143498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.129335    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gcd\" (UniqueName: \"kubernetes.io/projected/e7774cd9-8f15-45ad-9372-521c16e5a330-kube-api-access-s2gcd\") pod \"payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs\" (UID: \"e7774cd9-8f15-45ad-9372-521c16e5a330\") " pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:26.143498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.129426    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7774cd9-8f15-45ad-9372-521c16e5a330-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs\" (UID: \"e7774cd9-8f15-45ad-9372-521c16e5a330\") " pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:26.147152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.147116    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043d11d3-326a-4c67-817a-7b9a7cdf9596" path="/var/lib/kubelet/pods/043d11d3-326a-4c67-817a-7b9a7cdf9596/volumes"
Oct 30 22:30:26.147890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.147864    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"]
Oct 30 22:30:26.252243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.231562    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9glb\" (UniqueName: \"kubernetes.io/projected/9525c8e4-f28e-4563-9147-9d8cb7e01e02-kube-api-access-w9glb\") pod \"payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8\" (UID: \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:26.252243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.231682    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525c8e4-f28e-4563-9147-9d8cb7e01e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8\" (UID: \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:26.252243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.232028    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gcd\" (UniqueName: \"kubernetes.io/projected/e7774cd9-8f15-45ad-9372-521c16e5a330-kube-api-access-s2gcd\") pod \"payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs\" (UID: \"e7774cd9-8f15-45ad-9372-521c16e5a330\") " pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:26.276296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.253007    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7774cd9-8f15-45ad-9372-521c16e5a330-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs\" (UID: \"e7774cd9-8f15-45ad-9372-521c16e5a330\") " pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:26.276296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.271292    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7774cd9-8f15-45ad-9372-521c16e5a330-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs\" (UID: \"e7774cd9-8f15-45ad-9372-521c16e5a330\") " pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:26.276296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.275131    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gcd\" (UniqueName: \"kubernetes.io/projected/e7774cd9-8f15-45ad-9372-521c16e5a330-kube-api-access-s2gcd\") pod \"payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs\" (UID: \"e7774cd9-8f15-45ad-9372-521c16e5a330\") " pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:26.356260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.337813    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:26.372242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.359704    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9glb\" (UniqueName: \"kubernetes.io/projected/9525c8e4-f28e-4563-9147-9d8cb7e01e02-kube-api-access-w9glb\") pod \"payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8\" (UID: \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:26.372242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.359819    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525c8e4-f28e-4563-9147-9d8cb7e01e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8\" (UID: \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:26.372242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.372194    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525c8e4-f28e-4563-9147-9d8cb7e01e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8\" (UID: \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:26.451455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.403925    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9glb\" (UniqueName: \"kubernetes.io/projected/9525c8e4-f28e-4563-9147-9d8cb7e01e02-kube-api-access-w9glb\") pod \"payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8\" (UID: \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:26.519725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.519690    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:26.580552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.580514    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"]
Oct 30 22:30:26.591677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.591635    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.613801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.605833    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"]
Oct 30 22:30:26.673057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.672607    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-data\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.673057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.672651    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-certs\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.673057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.672713    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl5rc\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-kube-api-access-fl5rc\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.673057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.672746    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-credential-socket\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.673057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.672817    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-token\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.673057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.672864    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-socket\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.673057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.673022    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-podinfo\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.673564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.673193    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istiod-ca-cert\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.673564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.673297    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-envoy\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.758260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.758165    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bf9c2f-openshift-default-dockercfg-mjcmw\""
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.919984    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-data\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.920033    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-certs\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.920105    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl5rc\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-kube-api-access-fl5rc\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.920152    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-credential-socket\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.920311    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-token\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.920372    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-socket\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.920581    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-podinfo\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.920622    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istiod-ca-cert\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.920660    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-envoy\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.925434    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"]
Oct 30 22:30:26.947261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.945373    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-credential-socket\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.963291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.945800    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-data\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:26.965726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.958600    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-certs\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:27.040626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.958932    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-socket\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:27.040626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:26.959862    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istiod-ca-cert\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:27.040626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.032174    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-podinfo\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:27.041437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.041406    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-envoy\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:27.041635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.041486    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-token\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:27.042632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.042601    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl5rc\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-kube-api-access-fl5rc\") pod \"e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") " pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:27.117917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.117877    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"]
Oct 30 22:30:27.249523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.249494    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:27.690259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.689729    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs" event={"ID":"e7774cd9-8f15-45ad-9372-521c16e5a330","Type":"ContainerStarted","Data":"b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8"}
Oct 30 22:30:27.690259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.689804    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs" event={"ID":"e7774cd9-8f15-45ad-9372-521c16e5a330","Type":"ContainerStarted","Data":"d7c13e186664749981fb33259fff1c17827fcc6de437ed04bc7d28215ba290ed"}
Oct 30 22:30:27.691367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.690858    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:27.695244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.694050    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8" event={"ID":"9525c8e4-f28e-4563-9147-9d8cb7e01e02","Type":"ContainerStarted","Data":"f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f"}
Oct 30 22:30:27.695244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.694098    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8" event={"ID":"9525c8e4-f28e-4563-9147-9d8cb7e01e02","Type":"ContainerStarted","Data":"bcc124262e73bd64a632d89e1555fcf137017b6437f10c6c4534c2dda6745e44"}
Oct 30 22:30:27.695244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.694954    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:27.700566 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:27.700526    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb3f78a_618e_484e_b9ba_49ee3dd6c589.slice/crio-cc6a362cfd5b18fcefd4ee58ada5cc64da7dd9dfc791ef830fd7eeae9d18d9f2 WatchSource:0}: Error finding container cc6a362cfd5b18fcefd4ee58ada5cc64da7dd9dfc791ef830fd7eeae9d18d9f2: Status 404 returned error can't find the container with id cc6a362cfd5b18fcefd4ee58ada5cc64da7dd9dfc791ef830fd7eeae9d18d9f2
Oct 30 22:30:27.703867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.703765    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:27.704246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.704011    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:27.704246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.704096    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:27.706500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.706472    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"]
Oct 30 22:30:27.742345 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.742245    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs" podStartSLOduration=2.742209925 podStartE2EDuration="2.742209925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:27.737770327 +0000 UTC m=+1411.776709214" watchObservedRunningTime="2026-10-30 22:30:27.742209925 +0000 UTC m=+1411.781148813"
Oct 30 22:30:27.785518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:27.785452    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8" podStartSLOduration=1.785427196 podStartE2EDuration="1.785427196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:27.772709778 +0000 UTC m=+1411.811648665" watchObservedRunningTime="2026-10-30 22:30:27.785427196 +0000 UTC m=+1411.824366074"
Oct 30 22:30:28.707780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:28.707736    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" event={"ID":"dfb3f78a-618e-484e-b9ba-49ee3dd6c589","Type":"ContainerStarted","Data":"cdeac9cb3e79b8a74f179f3761b0a685825c6a5d4f00418a2d5ad6676e8e9648"}
Oct 30 22:30:28.708328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:28.707831    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" event={"ID":"dfb3f78a-618e-484e-b9ba-49ee3dd6c589","Type":"ContainerStarted","Data":"cc6a362cfd5b18fcefd4ee58ada5cc64da7dd9dfc791ef830fd7eeae9d18d9f2"}
Oct 30 22:30:28.807129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:28.807059    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podStartSLOduration=2.807035348 podStartE2EDuration="2.807035348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:28.755874955 +0000 UTC m=+1412.794813847" watchObservedRunningTime="2026-10-30 22:30:28.807035348 +0000 UTC m=+1412.845974235"
Oct 30 22:30:29.043717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:29.043622    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"]
Oct 30 22:30:29.057851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:29.057814    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d52998c-xj82m"]
Oct 30 22:30:29.250193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:29.250152    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:29.271380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:29.271352    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:29.892241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:29.888651    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:29.909270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:29.900721    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:30:30.108693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:30.108656    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07b41a9-3afc-443c-b38e-bb8fcd634f3c" path="/var/lib/kubelet/pods/f07b41a9-3afc-443c-b38e-bb8fcd634f3c/volumes"
Oct 30 22:30:38.728894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:38.727540    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:30:39.889263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:39.889207    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:30:40.106772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.106742    4581 generic.go:356] "Generic (PLEG): container finished" podID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerID="6051edd0e2f2dac4628d09b571d39054fb76b2ad37c7e8e90abd0082cd5ed4da" exitCode=0
Oct 30 22:30:40.106974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.106815    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" event={"ID":"c3c8d384-92f1-40f3-a233-84519c2ee72f","Type":"ContainerDied","Data":"6051edd0e2f2dac4628d09b571d39054fb76b2ad37c7e8e90abd0082cd5ed4da"}
Oct 30 22:30:40.107428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.107405    4581 scope.go:122] "RemoveContainer" containerID="6051edd0e2f2dac4628d09b571d39054fb76b2ad37c7e8e90abd0082cd5ed4da"
Oct 30 22:30:40.110980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.110926    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:40.111134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.111112    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:40.111262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.111242    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:40.594330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.592150    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"]
Oct 30 22:30:40.598497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.598467    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.613930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.606196    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"]
Oct 30 22:30:40.694464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.694418    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.694679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.694603    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79545781-49d9-4fba-be6c-25548aabcccf-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.694679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.694641    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.694810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.694706    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.694810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.694744    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.694810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.694771    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79545781-49d9-4fba-be6c-25548aabcccf-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.694950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.694821    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhr5l\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-kube-api-access-nhr5l\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.695038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.695016    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.695292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.695272    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.699404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.699382    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-370ec2-openshift-default-dockercfg-lcw59\""
Oct 30 22:30:40.796734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.796689    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.796927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.796790    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.796927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.796897    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797034    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79545781-49d9-4fba-be6c-25548aabcccf-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797060    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797078    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797103    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797138    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797164    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79545781-49d9-4fba-be6c-25548aabcccf-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797170    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797206    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhr5l\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-kube-api-access-nhr5l\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797207    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797600    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.797982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.797958    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79545781-49d9-4fba-be6c-25548aabcccf-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.800129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.800102    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79545781-49d9-4fba-be6c-25548aabcccf-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.803899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.803874    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.815395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.815323    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.816337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.816282    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhr5l\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-kube-api-access-nhr5l\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:40.822483 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:30:40.822453    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:30:40.885946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.885898    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:30:40.953800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:40.951560    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:41.140934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.140845    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" event={"ID":"c3c8d384-92f1-40f3-a233-84519c2ee72f","Type":"ContainerStarted","Data":"099bcf2d5fb570a442bff8b1a4cf223ea91a23a85672570d7ad33c2f19ad9114"}
Oct 30 22:30:41.409838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.409798    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"]
Oct 30 22:30:41.411468 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:41.411436    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79545781_49d9_4fba_be6c_25548aabcccf.slice/crio-201d3f2a9615f06a33151b345b78f8d8301e1d652f97b56e1048a36af3d1eccd WatchSource:0}: Error finding container 201d3f2a9615f06a33151b345b78f8d8301e1d652f97b56e1048a36af3d1eccd: Status 404 returned error can't find the container with id 201d3f2a9615f06a33151b345b78f8d8301e1d652f97b56e1048a36af3d1eccd
Oct 30 22:30:41.414611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.414577    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:41.414720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.414686    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:41.414791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.414736    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:41.541152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.541121    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:30:41.542560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.542526    4581 patch_prober.go:29] interesting pod/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:30:41.542696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.542585    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:30:41.645030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.636262    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"]
Oct 30 22:30:41.646047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.646012    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.712285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.697774    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7a1ee7-openshift-default-dockercfg-w28ch\""
Oct 30 22:30:41.712285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.702013    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"]
Oct 30 22:30:41.725873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.725838    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-envoy\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.726107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.726087    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31025e23-34e0-4096-88a1-85cd62392554-istio-podinfo\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.726296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.726275    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nw2\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-kube-api-access-q9nw2\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.726470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.726449    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-credential-socket\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.726592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.726576    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-certs\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.726741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.726726    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-istio-token\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.726857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.726844    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-data\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.727002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.726986    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-socket\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.727309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.727288    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31025e23-34e0-4096-88a1-85cd62392554-istiod-ca-cert\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.871762    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-envoy\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.871812    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31025e23-34e0-4096-88a1-85cd62392554-istio-podinfo\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.871866    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nw2\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-kube-api-access-q9nw2\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.871936    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-credential-socket\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.871980    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-certs\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.872058    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-istio-token\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.872101    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-data\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.872171    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-socket\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.872404    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31025e23-34e0-4096-88a1-85cd62392554-istiod-ca-cert\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.878272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.873324    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31025e23-34e0-4096-88a1-85cd62392554-istiod-ca-cert\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:41.912058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.912027    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-certs\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.011394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.916912    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-data\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.017243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.917433    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-socket\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.017505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.917651    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-credential-socket\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.017637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:41.953155    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-envoy\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.124242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.089939    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-istio-token\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.124242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.091810    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31025e23-34e0-4096-88a1-85cd62392554-istio-podinfo\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.224243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.215744    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nw2\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-kube-api-access-q9nw2\") pod \"e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") " pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.319562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.319465    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" event={"ID":"79545781-49d9-4fba-be6c-25548aabcccf","Type":"ContainerStarted","Data":"44a5cd5113bf760b10507a789ecfd5cc36fdad2b442cddb0300aff685c4fc5a0"}
Oct 30 22:30:42.319562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.319529    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" event={"ID":"79545781-49d9-4fba-be6c-25548aabcccf","Type":"ContainerStarted","Data":"201d3f2a9615f06a33151b345b78f8d8301e1d652f97b56e1048a36af3d1eccd"}
Oct 30 22:30:42.338484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.338454    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:42.390313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.375388    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podStartSLOduration=2.375369562 podStartE2EDuration="2.375369562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:42.374893059 +0000 UTC m=+1426.413831946" watchObservedRunningTime="2026-10-30 22:30:42.375369562 +0000 UTC m=+1426.414308448"
Oct 30 22:30:42.536287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.536252    4581 patch_prober.go:29] interesting pod/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:30:42.536484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.536307    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:30:42.791001 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:42.790966    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31025e23_34e0_4096_88a1_85cd62392554.slice/crio-19510a77daa263ebf496a86854234dc472a32741f33a3c8515276fe0a7a4f19a WatchSource:0}: Error finding container 19510a77daa263ebf496a86854234dc472a32741f33a3c8515276fe0a7a4f19a: Status 404 returned error can't find the container with id 19510a77daa263ebf496a86854234dc472a32741f33a3c8515276fe0a7a4f19a
Oct 30 22:30:42.792450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.792423    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"]
Oct 30 22:30:42.797256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.794322    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:42.797256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.794437    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:42.797256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.794468    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:43.017243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:42.969377    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:43.017243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:43.010460    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:43.017243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:43.010614    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 22:30:43.386245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:43.379644    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" event={"ID":"31025e23-34e0-4096-88a1-85cd62392554","Type":"ContainerStarted","Data":"0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040"}
Oct 30 22:30:43.386245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:43.379708    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" event={"ID":"31025e23-34e0-4096-88a1-85cd62392554","Type":"ContainerStarted","Data":"19510a77daa263ebf496a86854234dc472a32741f33a3c8515276fe0a7a4f19a"}
Oct 30 22:30:43.535796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:43.534849    4581 patch_prober.go:29] interesting pod/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:30:43.535796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:43.534910    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:30:43.961244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:43.956266    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:43.961244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:43.956347    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 22:30:44.339041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.338947    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:44.347324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.347298    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:44.425091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.425058    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:44.426439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.426415    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:30:44.428768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.428722    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podStartSLOduration=3.42870834 podStartE2EDuration="3.42870834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:43.437038416 +0000 UTC m=+1427.475977303" watchObservedRunningTime="2026-10-30 22:30:44.42870834 +0000 UTC m=+1428.467647222"
Oct 30 22:30:44.541280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.534921    4581 patch_prober.go:29] interesting pod/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:30:44.541280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.534994    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:30:44.928272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.925543    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"]
Oct 30 22:30:44.933985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.933712    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:44.944683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.944649    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"]
Oct 30 22:30:45.035465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.949751    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-d9f1c2-openshift-default-dockercfg-rvd79\""
Oct 30 22:30:45.035740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:44.987690    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.035928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.035899    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 22:30:45.036051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.031429    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.036247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.036207    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4b4443-3f69-43a0-9a55-118b93e68702-istiod-ca-cert\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.036503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.036480    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-certs\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.036665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.036647    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-data\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.036847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.036822    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-envoy\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.037081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.037062    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7kv2\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-kube-api-access-v7kv2\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.037266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.037248    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-podinfo\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.037460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.037444    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-token\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.037653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.037636    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-credential-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.183535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.183425    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4b4443-3f69-43a0-9a55-118b93e68702-istiod-ca-cert\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.183910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.183886    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-certs\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.210391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.210338    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-data\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.210708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.210688    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-envoy\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.211023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.211005    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7kv2\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-kube-api-access-v7kv2\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.218245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.211165    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-podinfo\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.226821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.218555    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-token\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.252246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.227155    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-credential-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.257506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.185137    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-certs\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.257506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.184877    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4b4443-3f69-43a0-9a55-118b93e68702-istiod-ca-cert\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.258169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.258105    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-credential-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.258604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.258580    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-data\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.273253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.268818    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.273253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.269289    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-socket\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.436307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.426018    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7kv2\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-kube-api-access-v7kv2\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.519111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.464663    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-token\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.549244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.545998    4581 patch_prober.go:29] interesting pod/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.549244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.546052    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:30:45.580242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.579148    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-envoy\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.580242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.579759    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-podinfo\") pod \"e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") " pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.868597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.867402    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:45.956063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.954641    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.956063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:45.954703    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 22:30:46.829876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.829821    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:46.830543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.830507    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:46.830702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.830684    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:30:46.840023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.839848    4581 patch_prober.go:29] interesting pod/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:30:46.840023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.839896    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:30:46.843096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.842379    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"]
Oct 30 22:30:46.849733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.849704    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" event={"ID":"ae4b4443-3f69-43a0-9a55-118b93e68702","Type":"ContainerStarted","Data":"3f6d3670f06dac0b25f3c50808293f2aee72bfe40bc5522a0766c19179248989"}
Oct 30 22:30:46.959454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.959418    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:46.959746 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:46.959717    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 22:30:47.534150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:47.534111    4581 patch_prober.go:29] interesting pod/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:30:47.534344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:47.534176    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:30:47.895835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:47.894065    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" event={"ID":"ae4b4443-3f69-43a0-9a55-118b93e68702","Type":"ContainerStarted","Data":"1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8"}
Oct 30 22:30:47.954247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:47.950788    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podStartSLOduration=3.950762317 podStartE2EDuration="3.950762317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:47.942259144 +0000 UTC m=+1431.981198036" watchObservedRunningTime="2026-10-30 22:30:47.950762317 +0000 UTC m=+1431.989701206"
Oct 30 22:30:47.966245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:47.956647    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:47.966245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:47.956710    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 22:30:48.663080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.661134    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:30:48.663080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.662918    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:30:48.670897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.670864    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:30:48.873470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.873381    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:48.904524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.903856    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:48.915992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.915885    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:48.917929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.917901    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:30:48.955727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.955694    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:48.956025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:48.955989    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 22:30:49.960482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:49.960456    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:49.961025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:49.961000    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:49.961852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:49.961833    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:30:56.317398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.317356    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"]
Oct 30 22:30:56.323310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.323282    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:56.342858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.342826    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"]
Oct 30 22:30:56.346406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.346379    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3d38f6\""
Oct 30 22:30:56.393425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.393384    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"]
Oct 30 22:30:56.398328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.398301    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:56.405769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.405734    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c90a6-d048-460f-92b4-bb81e0ea69c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9\" (UID: \"850c90a6-d048-460f-92b4-bb81e0ea69c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:56.405956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.405795    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrj5w\" (UniqueName: \"kubernetes.io/projected/213b7946-13df-4673-97bd-e8fe3fde5d80-kube-api-access-rrj5w\") pod \"payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42\" (UID: \"213b7946-13df-4673-97bd-e8fe3fde5d80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:56.406074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.406036    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213b7946-13df-4673-97bd-e8fe3fde5d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42\" (UID: \"213b7946-13df-4673-97bd-e8fe3fde5d80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:56.406258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.406117    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmjg\" (UniqueName: \"kubernetes.io/projected/850c90a6-d048-460f-92b4-bb81e0ea69c5-kube-api-access-8vmjg\") pod \"payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9\" (UID: \"850c90a6-d048-460f-92b4-bb81e0ea69c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:56.411290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.411263    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3d38f6-dockercfg-ljs8n\""
Oct 30 22:30:56.419454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.419379    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"]
Oct 30 22:30:56.742445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.742409    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c90a6-d048-460f-92b4-bb81e0ea69c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9\" (UID: \"850c90a6-d048-460f-92b4-bb81e0ea69c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:56.764240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.746314    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrj5w\" (UniqueName: \"kubernetes.io/projected/213b7946-13df-4673-97bd-e8fe3fde5d80-kube-api-access-rrj5w\") pod \"payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42\" (UID: \"213b7946-13df-4673-97bd-e8fe3fde5d80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:56.774247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.764762    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213b7946-13df-4673-97bd-e8fe3fde5d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42\" (UID: \"213b7946-13df-4673-97bd-e8fe3fde5d80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:56.788698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.778673    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213b7946-13df-4673-97bd-e8fe3fde5d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42\" (UID: \"213b7946-13df-4673-97bd-e8fe3fde5d80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:56.788943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.787570    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c90a6-d048-460f-92b4-bb81e0ea69c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9\" (UID: \"850c90a6-d048-460f-92b4-bb81e0ea69c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:56.789070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.788696    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmjg\" (UniqueName: \"kubernetes.io/projected/850c90a6-d048-460f-92b4-bb81e0ea69c5-kube-api-access-8vmjg\") pod \"payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9\" (UID: \"850c90a6-d048-460f-92b4-bb81e0ea69c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:56.820239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.818810    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrj5w\" (UniqueName: \"kubernetes.io/projected/213b7946-13df-4673-97bd-e8fe3fde5d80-kube-api-access-rrj5w\") pod \"payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42\" (UID: \"213b7946-13df-4673-97bd-e8fe3fde5d80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:56.825836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.825801    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmjg\" (UniqueName: \"kubernetes.io/projected/850c90a6-d048-460f-92b4-bb81e0ea69c5-kube-api-access-8vmjg\") pod \"payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9\" (UID: \"850c90a6-d048-460f-92b4-bb81e0ea69c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:56.938934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:56.938895    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:57.024105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:57.024080    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:57.283643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:57.283611    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"]
Oct 30 22:30:57.285184 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:57.285156    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850c90a6_d048_460f_92b4_bb81e0ea69c5.slice/crio-ec1dae9c2e80ac7c8847afffe7dc6b211763832b56e10094ef8efebfd7e7118a WatchSource:0}: Error finding container ec1dae9c2e80ac7c8847afffe7dc6b211763832b56e10094ef8efebfd7e7118a: Status 404 returned error can't find the container with id ec1dae9c2e80ac7c8847afffe7dc6b211763832b56e10094ef8efebfd7e7118a
Oct 30 22:30:57.364412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:57.364388    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"]
Oct 30 22:30:57.367028 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:57.366997    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213b7946_13df_4673_97bd_e8fe3fde5d80.slice/crio-04ffe653a278ce9fa0fa96de6f59fb4573bda5c2bb99cb98b275a08638a48949 WatchSource:0}: Error finding container 04ffe653a278ce9fa0fa96de6f59fb4573bda5c2bb99cb98b275a08638a48949: Status 404 returned error can't find the container with id 04ffe653a278ce9fa0fa96de6f59fb4573bda5c2bb99cb98b275a08638a48949
Oct 30 22:30:58.027880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.027841    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9" event={"ID":"850c90a6-d048-460f-92b4-bb81e0ea69c5","Type":"ContainerStarted","Data":"7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166"}
Oct 30 22:30:58.028099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.027907    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9" event={"ID":"850c90a6-d048-460f-92b4-bb81e0ea69c5","Type":"ContainerStarted","Data":"ec1dae9c2e80ac7c8847afffe7dc6b211763832b56e10094ef8efebfd7e7118a"}
Oct 30 22:30:58.028099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.027937    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:30:58.029750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.029721    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42" event={"ID":"213b7946-13df-4673-97bd-e8fe3fde5d80","Type":"ContainerStarted","Data":"54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66"}
Oct 30 22:30:58.029907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.029767    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42" event={"ID":"213b7946-13df-4673-97bd-e8fe3fde5d80","Type":"ContainerStarted","Data":"04ffe653a278ce9fa0fa96de6f59fb4573bda5c2bb99cb98b275a08638a48949"}
Oct 30 22:30:58.029907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.029795    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:30:58.062057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.061997    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9" podStartSLOduration=2.061983788 podStartE2EDuration="2.061983788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:58.061542548 +0000 UTC m=+1442.100481434" watchObservedRunningTime="2026-10-30 22:30:58.061983788 +0000 UTC m=+1442.100922673"
Oct 30 22:30:58.097235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.097143    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42" podStartSLOduration=2.097127145 podStartE2EDuration="2.097127145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:58.096482198 +0000 UTC m=+1442.135421085" watchObservedRunningTime="2026-10-30 22:30:58.097127145 +0000 UTC m=+1442.136066032"
Oct 30 22:30:58.339351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.339262    4581 scope.go:122] "RemoveContainer" containerID="6916c8709c50fce9a6b94200ac688d7d2804a153b07bc3545b0719f89b729cce"
Oct 30 22:30:58.352411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.352387    4581 scope.go:122] "RemoveContainer" containerID="97ab4d1c8410ab3036e2712361c2ec0bc30b1f1ef741fa39de46c3a358147333"
Oct 30 22:30:58.366896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.366869    4581 scope.go:122] "RemoveContainer" containerID="d72569e505cde1ea477941ab28b929234ef06234e2bc16fdf8d33caf747bdbf7"
Oct 30 22:30:58.824455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.824420    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"]
Oct 30 22:30:58.831060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.831037    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:58.849004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.848979    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"]
Oct 30 22:30:58.852156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.852132    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:58.852323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.852140    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:58.852323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.852245    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:58.852886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.852860    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:30:58.937917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.937878    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwp6q\" (UniqueName: \"kubernetes.io/projected/10e9190e-52a3-49b3-9230-551d261c2eaf-kube-api-access-mwp6q\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:58.938116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.938093    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:58.938296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:58.938274    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:59.039554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:59.039517    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwp6q\" (UniqueName: \"kubernetes.io/projected/10e9190e-52a3-49b3-9230-551d261c2eaf-kube-api-access-mwp6q\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:59.039733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:59.039633    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:59.039733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:59.039708    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:59.040324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:59.040302    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:59.042345 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:59.042319    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:59.057118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:59.057093    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwp6q\" (UniqueName: \"kubernetes.io/projected/10e9190e-52a3-49b3-9230-551d261c2eaf-kube-api-access-mwp6q\") pod \"maas-api-revoke-keys-e2e-mt-2949c143-4nwbv\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:59.152780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:59.152746    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:30:59.480355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:30:59.480322    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"]
Oct 30 22:30:59.481245 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:30:59.481198    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e9190e_52a3_49b3_9230_551d261c2eaf.slice/crio-2a12ca3c8de5eaa8002f7b705f02ff105c2f1cb45cf5327aca3c8cda87e3678a WatchSource:0}: Error finding container 2a12ca3c8de5eaa8002f7b705f02ff105c2f1cb45cf5327aca3c8cda87e3678a: Status 404 returned error can't find the container with id 2a12ca3c8de5eaa8002f7b705f02ff105c2f1cb45cf5327aca3c8cda87e3678a
Oct 30 22:31:00.049502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:00.049466    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv" event={"ID":"10e9190e-52a3-49b3-9230-551d261c2eaf","Type":"ContainerStarted","Data":"a5540e9df7235a724518302c7f20fafa629ff47502561b1bdeaf4fa5ed18010e"}
Oct 30 22:31:00.049693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:00.049550    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv" event={"ID":"10e9190e-52a3-49b3-9230-551d261c2eaf","Type":"ContainerStarted","Data":"2a12ca3c8de5eaa8002f7b705f02ff105c2f1cb45cf5327aca3c8cda87e3678a"}
Oct 30 22:31:00.084627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:00.084565    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv" podStartSLOduration=2.084547042 podStartE2EDuration="2.084547042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:00.083779213 +0000 UTC m=+1444.122718099" watchObservedRunningTime="2026-10-30 22:31:00.084547042 +0000 UTC m=+1444.123485931"
Oct 30 22:31:01.062659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:01.062623    4581 generic.go:356] "Generic (PLEG): container finished" podID="10e9190e-52a3-49b3-9230-551d261c2eaf" containerID="a5540e9df7235a724518302c7f20fafa629ff47502561b1bdeaf4fa5ed18010e" exitCode=0
Oct 30 22:31:01.063080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:01.062713    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv" event={"ID":"10e9190e-52a3-49b3-9230-551d261c2eaf","Type":"ContainerDied","Data":"a5540e9df7235a724518302c7f20fafa629ff47502561b1bdeaf4fa5ed18010e"}
Oct 30 22:31:02.271728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.271701    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:31:02.405561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.405525    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e9190e-52a3-49b3-9230-551d261c2eaf-kube-api-access-mwp6q\" (UniqueName: \"kubernetes.io/projected/10e9190e-52a3-49b3-9230-551d261c2eaf-kube-api-access-mwp6q\") pod \"10e9190e-52a3-49b3-9230-551d261c2eaf\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") "
Oct 30 22:31:02.405561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.405568    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-xks-ca\") pod \"10e9190e-52a3-49b3-9230-551d261c2eaf\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") "
Oct 30 22:31:02.405791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.405633    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-openshift-ca\") pod \"10e9190e-52a3-49b3-9230-551d261c2eaf\" (UID: \"10e9190e-52a3-49b3-9230-551d261c2eaf\") "
Oct 30 22:31:02.406056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.406024    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-openshift-ca" pod "10e9190e-52a3-49b3-9230-551d261c2eaf" (UID: "10e9190e-52a3-49b3-9230-551d261c2eaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:02.406598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.406576    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:02.407895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.407873    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-xks-ca" pod "10e9190e-52a3-49b3-9230-551d261c2eaf" (UID: "10e9190e-52a3-49b3-9230-551d261c2eaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:02.408013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.407996    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e9190e-52a3-49b3-9230-551d261c2eaf-kube-api-access-mwp6q" pod "10e9190e-52a3-49b3-9230-551d261c2eaf" (UID: "10e9190e-52a3-49b3-9230-551d261c2eaf"). InnerVolumeSpecName "kube-api-access-mwp6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:02.508149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.508109    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwp6q\" (UniqueName: \"kubernetes.io/projected/10e9190e-52a3-49b3-9230-551d261c2eaf-kube-api-access-mwp6q\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:02.508149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:02.508140    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e9190e-52a3-49b3-9230-551d261c2eaf-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:03.090567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:03.090525    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv" event={"ID":"10e9190e-52a3-49b3-9230-551d261c2eaf","Type":"ContainerDied","Data":"2a12ca3c8de5eaa8002f7b705f02ff105c2f1cb45cf5327aca3c8cda87e3678a"}
Oct 30 22:31:03.090567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:03.090572    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a12ca3c8de5eaa8002f7b705f02ff105c2f1cb45cf5327aca3c8cda87e3678a"
Oct 30 22:31:03.090846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:03.090616    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"
Oct 30 22:31:09.026703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:09.026653    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"]
Oct 30 22:31:09.041766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:09.041738    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:31:09.041931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:09.041804    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:31:09.293371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:09.293274    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"]
Oct 30 22:31:09.293561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:09.293475    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy" containerID="cri-o://03ce93aa199cc0c0a462fc51b9996e0fe14615921ca7e960134b5ef94e7db054" gracePeriod=30
Oct 30 22:31:10.660733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.660693    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"]
Oct 30 22:31:10.662759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.662725    4581 cpu_manager.go:414] "containerMap: removing container" podUID="10e9190e-52a3-49b3-9230-551d261c2eaf" containerName="revoke-keys"
Oct 30 22:31:10.663038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.663017    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e9190e-52a3-49b3-9230-551d261c2eaf" containerName="revoke-keys"
Oct 30 22:31:10.663854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.663470    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e9190e-52a3-49b3-9230-551d261c2eaf" containerName="revoke-keys"
Oct 30 22:31:10.670984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.670956    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.710286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.706891    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"]
Oct 30 22:31:10.716195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.716150    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.716513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.716274    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.716513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.716338    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.716513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.716368    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.716704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.716562    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cssdr\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-kube-api-access-cssdr\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.716704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.716685    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.716829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.716715    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.716829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.716793    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.717045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.717001    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.717496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.717475    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.718337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.718312    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"]
Oct 30 22:31:10.730471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.730438    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"]
Oct 30 22:31:10.819408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819364    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cssdr\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-kube-api-access-cssdr\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.819652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819489    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.819652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819571    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.819652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819623    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.819652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819650    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e5b32e-12f7-4449-887b-e73e9b332b63-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.819876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819735    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.819876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819790    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.819876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819864    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.820044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819888    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.820044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819974    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.820044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.819998    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-kube-api-access-qs6qt\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.820208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.820087    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.820208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.820119    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.820208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.820145    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.820208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.820184    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.820208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.820244    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.820557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.820279    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.821058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.820761    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.821058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.820838    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.821398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.821117    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.821398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.821356    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.821543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.821443    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.821599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.821579    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.823977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.823950    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.825212 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.825182    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.829534 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:31:10.829505    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:31:10.837369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.837338    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.837627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.837608    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cssdr\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-kube-api-access-cssdr\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:10.923495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923389    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-kube-api-access-qs6qt\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.923495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923482    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.923753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923513    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.923753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923548    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.923753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923645    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.923907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923814    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.923907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923882    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e5b32e-12f7-4449-887b-e73e9b332b63-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.924007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923970    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.924007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923988    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.924007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.923993    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.924169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.924079    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.924340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.924316    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.924563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.924541    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.924654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.924637    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e5b32e-12f7-4449-887b-e73e9b332b63-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.926378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.926355    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.926550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.926523    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.940593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.940562    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:10.940775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:10.940741    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-kube-api-access-qs6qt\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:11.087363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.087329    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:11.094356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.094330    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:11.441277 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:31:11.441242    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34b75f1_c64b_444d_94ad_c5f7798bf66f.slice/crio-1894b3b2f94437a2aab9384ab7a485a9fbfd27cac2cd336430892f310bd72291 WatchSource:0}: Error finding container 1894b3b2f94437a2aab9384ab7a485a9fbfd27cac2cd336430892f310bd72291: Status 404 returned error can't find the container with id 1894b3b2f94437a2aab9384ab7a485a9fbfd27cac2cd336430892f310bd72291
Oct 30 22:31:11.441644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.441622    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"]
Oct 30 22:31:11.444662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.444626    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:31:11.444760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.444695    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:31:11.444760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.444723    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:31:11.452584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.448520    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"]
Oct 30 22:31:11.458254 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:31:11.458194    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e5b32e_12f7_4449_887b_e73e9b332b63.slice/crio-778d90107c7dc111d87c17926fcd7560e571df5c608d9c9ce1b3d6ee5ed1fd44 WatchSource:0}: Error finding container 778d90107c7dc111d87c17926fcd7560e571df5c608d9c9ce1b3d6ee5ed1fd44: Status 404 returned error can't find the container with id 778d90107c7dc111d87c17926fcd7560e571df5c608d9c9ce1b3d6ee5ed1fd44
Oct 30 22:31:11.466458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.466420    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:31:11.466588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.466511    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:31:11.466588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:11.466563    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:31:12.412728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:12.412692    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" event={"ID":"f34b75f1-c64b-444d-94ad-c5f7798bf66f","Type":"ContainerStarted","Data":"208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1"}
Oct 30 22:31:12.413137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:12.412755    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" event={"ID":"f34b75f1-c64b-444d-94ad-c5f7798bf66f","Type":"ContainerStarted","Data":"1894b3b2f94437a2aab9384ab7a485a9fbfd27cac2cd336430892f310bd72291"}
Oct 30 22:31:12.415165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:12.415138    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" event={"ID":"32e5b32e-12f7-4449-887b-e73e9b332b63","Type":"ContainerStarted","Data":"b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c"}
Oct 30 22:31:12.415377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:12.415361    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" event={"ID":"32e5b32e-12f7-4449-887b-e73e9b332b63","Type":"ContainerStarted","Data":"778d90107c7dc111d87c17926fcd7560e571df5c608d9c9ce1b3d6ee5ed1fd44"}
Oct 30 22:31:12.491244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:12.483854    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" podStartSLOduration=2.483834774 podStartE2EDuration="2.483834774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:12.469705438 +0000 UTC m=+1456.508644336" watchObservedRunningTime="2026-10-30 22:31:12.483834774 +0000 UTC m=+1456.522773662"
Oct 30 22:31:12.599273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:12.595683    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" podStartSLOduration=2.595656969 podStartE2EDuration="2.595656969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:12.534268451 +0000 UTC m=+1456.573207339" watchObservedRunningTime="2026-10-30 22:31:12.595656969 +0000 UTC m=+1456.634595859"
Oct 30 22:31:13.107599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:13.107564    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:13.107936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:13.107918    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:13.108621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:13.108594    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:13.108826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:13.108800    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:31:13.109163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:13.109125    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:31:13.109295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:13.109192    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:31:14.027121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:14.027064    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8" podUID="af752c40-842e-4c32-990f-da63c2a0066f" containerName="payload-pre-processing" containerID="cri-o://dd708a2058bf4dca6b0a7d7802179ceb96ee349064f1b0b67f975b518d036e54" gracePeriod=25
Oct 30 22:31:14.111245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:14.096726    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:14.111245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:14.096787    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:31:14.111245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:14.097426    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:31:14.111245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:14.097468    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:31:14.524247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:14.521642    4581 generic.go:356] "Generic (PLEG): container finished" podID="af752c40-842e-4c32-990f-da63c2a0066f" containerID="dd708a2058bf4dca6b0a7d7802179ceb96ee349064f1b0b67f975b518d036e54" exitCode=0
Oct 30 22:31:14.524247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:14.523089    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8" event={"ID":"af752c40-842e-4c32-990f-da63c2a0066f","Type":"ContainerDied","Data":"dd708a2058bf4dca6b0a7d7802179ceb96ee349064f1b0b67f975b518d036e54"}
Oct 30 22:31:15.123240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.119760    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:15.123240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.119822    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:31:15.168248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.132621    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:31:15.168248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.132685    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:31:15.412461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.411962    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:31:15.524250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.509977    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af752c40-842e-4c32-990f-da63c2a0066f-kube-api-access-xmkrm\" (UniqueName: \"kubernetes.io/projected/af752c40-842e-4c32-990f-da63c2a0066f-kube-api-access-xmkrm\") pod \"af752c40-842e-4c32-990f-da63c2a0066f\" (UID: \"af752c40-842e-4c32-990f-da63c2a0066f\") "
Oct 30 22:31:15.524250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.510087    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af752c40-842e-4c32-990f-da63c2a0066f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af752c40-842e-4c32-990f-da63c2a0066f-plugins-config-volume\") pod \"af752c40-842e-4c32-990f-da63c2a0066f\" (UID: \"af752c40-842e-4c32-990f-da63c2a0066f\") "
Oct 30 22:31:15.524250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.511611    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af752c40-842e-4c32-990f-da63c2a0066f-plugins-config-volume" pod "af752c40-842e-4c32-990f-da63c2a0066f" (UID: "af752c40-842e-4c32-990f-da63c2a0066f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:15.532304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.527954    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af752c40-842e-4c32-990f-da63c2a0066f-kube-api-access-xmkrm" pod "af752c40-842e-4c32-990f-da63c2a0066f" (UID: "af752c40-842e-4c32-990f-da63c2a0066f"). InnerVolumeSpecName "kube-api-access-xmkrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.545753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.543562    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8" event={"ID":"af752c40-842e-4c32-990f-da63c2a0066f","Type":"ContainerDied","Data":"3849b90a4c0032838863f95aceceb012e9c6c0cbd13742ed804630b41a92669f"}
Oct 30 22:31:15.545753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.543623    4581 scope.go:122] "RemoveContainer" containerID="dd708a2058bf4dca6b0a7d7802179ceb96ee349064f1b0b67f975b518d036e54"
Oct 30 22:31:15.545753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.543748    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"
Oct 30 22:31:15.618901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.616199    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmkrm\" (UniqueName: \"kubernetes.io/projected/af752c40-842e-4c32-990f-da63c2a0066f-kube-api-access-xmkrm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.619161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.619110    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af752c40-842e-4c32-990f-da63c2a0066f-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.619161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.618788    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" event={"ID":"5467bf2c-3a86-4dc1-9975-381cbb821574","Type":"ContainerDied","Data":"03ce93aa199cc0c0a462fc51b9996e0fe14615921ca7e960134b5ef94e7db054"}
Oct 30 22:31:15.619393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.618757    4581 generic.go:356] "Generic (PLEG): container finished" podID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerID="03ce93aa199cc0c0a462fc51b9996e0fe14615921ca7e960134b5ef94e7db054" exitCode=0
Oct 30 22:31:15.736297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.734536    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"]
Oct 30 22:31:15.744523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.743906    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2949c143-5f67f57d8c-pw4d8"]
Oct 30 22:31:15.823287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.816428    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:31:15.845828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.845784    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-socket\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.846039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.845878    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5467bf2c-3a86-4dc1-9975-381cbb821574-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5467bf2c-3a86-4dc1-9975-381cbb821574-istiod-ca-cert\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.846039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.845914    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-credential-socket\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.846039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.845964    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-data\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.846039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.846006    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-token\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-token\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.846039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.846029    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-podinfo\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.846422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.846053    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-certs\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.846422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.846134    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-kube-api-access-h2p74\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-kube-api-access-h2p74\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.846422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.846297    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-data" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.846921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.846899    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.847092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.847062    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-certs" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.847260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.847229    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-credential-socket" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.847446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.847410    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5467bf2c-3a86-4dc1-9975-381cbb821574-istiod-ca-cert" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:15.847634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.847613    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-socket" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.849293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.849267    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-podinfo" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:15.851057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.851022    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-kube-api-access-h2p74" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "kube-api-access-h2p74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.859735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.859681    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-token" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.947829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.947795    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-envoy\") pod \"5467bf2c-3a86-4dc1-9975-381cbb821574\" (UID: \"5467bf2c-3a86-4dc1-9975-381cbb821574\") "
Oct 30 22:31:15.948731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.948707    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.948866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.948732    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5467bf2c-3a86-4dc1-9975-381cbb821574-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.948866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.948748    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.948866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.948760    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.948866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.948773    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.948866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.948786    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.948866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.948798    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2p74\" (UniqueName: \"kubernetes.io/projected/5467bf2c-3a86-4dc1-9975-381cbb821574-kube-api-access-h2p74\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.951031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:15.951000    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-envoy" pod "5467bf2c-3a86-4dc1-9975-381cbb821574" (UID: "5467bf2c-3a86-4dc1-9975-381cbb821574"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.174305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.150782    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5467bf2c-3a86-4dc1-9975-381cbb821574-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:31:16.174305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.152154    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:16.174305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.152261    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:31:16.233244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.201418    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:31:16.233244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.201485    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:31:16.310246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.309661    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af752c40-842e-4c32-990f-da63c2a0066f" path="/var/lib/kubelet/pods/af752c40-842e-4c32-990f-da63c2a0066f/volumes"
Oct 30 22:31:16.935070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.935038    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97" event={"ID":"5467bf2c-3a86-4dc1-9975-381cbb821574","Type":"ContainerDied","Data":"31bced79189145adf5efb30716b8e14765f07ae7627cc2c3b61ef6dfb5593c10"}
Oct 30 22:31:16.935495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.935474    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"
Oct 30 22:31:16.935712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:16.935687    4581 scope.go:122] "RemoveContainer" containerID="03ce93aa199cc0c0a462fc51b9996e0fe14615921ca7e960134b5ef94e7db054"
Oct 30 22:31:17.037601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:17.037563    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"]
Oct 30 22:31:17.046119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:17.046065    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-ssk97"]
Oct 30 22:31:17.097499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:17.097460    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:17.097763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:17.097739    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:31:17.098551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:17.098526    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:31:17.098738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:17.098718    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:31:18.106239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:18.105320    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:18.106239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:18.105384    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:31:18.106239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:18.105815    4581 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:31:18.106239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:18.105845    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:31:18.131438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:18.131398    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" path="/var/lib/kubelet/pods/5467bf2c-3a86-4dc1-9975-381cbb821574/volumes"
Oct 30 22:31:19.094574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:19.094535    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:19.094870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:19.094846    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:19.095675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:19.095650    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:31:19.099681 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:19.099658    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:19.100054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:19.100031    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:19.100782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:19.100763    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:31:35.133718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:35.133676    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"]
Oct 30 22:31:35.139544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:35.139511    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2949c143-4nwbv"]
Oct 30 22:31:36.113525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:36.113485    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e9190e-52a3-49b3-9230-551d261c2eaf" path="/var/lib/kubelet/pods/10e9190e-52a3-49b3-9230-551d261c2eaf/volumes"
Oct 30 22:31:40.824386 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:31:40.824349    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:31:54.970134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:54.970081    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"]
Oct 30 22:31:54.977942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:54.977902    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy" containerID="cri-o://afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27" gracePeriod=30
Oct 30 22:31:56.935319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:56.935287    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:31:56.942480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:56.942451    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:31:58.510255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:58.510197    4581 scope.go:122] "RemoveContainer" containerID="397917c9499f49344afe0572cf91608efac089de861eb4dda072485a37d53526"
Oct 30 22:31:58.524405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:58.524379    4581 scope.go:122] "RemoveContainer" containerID="98369fc84b57406ed73afe90d10bb48fdeab304dc192a50c4e782e239a7a238e"
Oct 30 22:31:58.536335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:31:58.536314    4581 scope.go:122] "RemoveContainer" containerID="21fbdacfdd78c56c4ad066a327c4a8991f07e915a3adc27bdb0aa9a225c589a4"
Oct 30 22:32:00.564538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.564499    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:32:00.616261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616204    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-data\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616309    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-credential-socket\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616438    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-certs\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616492    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-socket\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616549    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-token\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-token\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616583    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-data" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:00.616736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616655    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istiod-ca-cert\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616690    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-credential-socket" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:00.616736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616704    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-kube-api-access-6sfqc\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-kube-api-access-6sfqc\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616763    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-podinfo\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616854    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-envoy\") pod \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\" (UID: \"74d3182d-81a5-4591-9a6d-ce1bfffeb438\") "
Oct 30 22:32:00.616960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.616854    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-certs" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:00.617332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.617303    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istiod-ca-cert" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:00.617528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.617445    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-socket" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:00.617927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.617868    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.617927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.617899    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.617927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.617917    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.618149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.617932    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.618149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.617948    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.619315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.619292    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-token" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:00.619532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.619503    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-envoy" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:00.619677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.619659    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-kube-api-access-6sfqc" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "kube-api-access-6sfqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:00.619932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.619914    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-podinfo" pod "74d3182d-81a5-4591-9a6d-ce1bfffeb438" (UID: "74d3182d-81a5-4591-9a6d-ce1bfffeb438"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:00.720162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.720120    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfqc\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-kube-api-access-6sfqc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.720162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.720167    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.720461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.720184    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.720461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:00.720202    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d3182d-81a5-4591-9a6d-ce1bfffeb438-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:01.307622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.307582    4581 generic.go:356] "Generic (PLEG): container finished" podID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerID="afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27" exitCode=0
Oct 30 22:32:01.307828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.307634    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"
Oct 30 22:32:01.307828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.307677    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" event={"ID":"74d3182d-81a5-4591-9a6d-ce1bfffeb438","Type":"ContainerDied","Data":"afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27"}
Oct 30 22:32:01.307828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.307738    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw" event={"ID":"74d3182d-81a5-4591-9a6d-ce1bfffeb438","Type":"ContainerDied","Data":"2a2e766ca17b034b87f0edb1c546ec02a5dd403e1c78f71922fbaae08a2d5460"}
Oct 30 22:32:01.307828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.307771    4581 scope.go:122] "RemoveContainer" containerID="afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27"
Oct 30 22:32:01.365165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.365126    4581 scope.go:122] "RemoveContainer" containerID="afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27"
Oct 30 22:32:01.365595 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:32:01.365569    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27\": container with ID starting with afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27 not found: ID does not exist" containerID="afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27"
Oct 30 22:32:01.365706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.365609    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27"} err="failed to get container status \"afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27\": rpc error: code = NotFound desc = could not find container \"afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27\": container with ID starting with afe9dc5304b4774c06faaca151c845249f3f825693551d988abddf98cdc48f27 not found: ID does not exist"
Oct 30 22:32:01.385236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.385184    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"]
Oct 30 22:32:01.394325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:01.394251    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-5zllw"]
Oct 30 22:32:02.101666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:02.101632    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" path="/var/lib/kubelet/pods/74d3182d-81a5-4591-9a6d-ce1bfffeb438/volumes"
Oct 30 22:32:04.610276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.610236    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"]
Oct 30 22:32:04.611963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.611936    4581 cpu_manager.go:414] "containerMap: removing container" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy"
Oct 30 22:32:04.612138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.612125    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy"
Oct 30 22:32:04.612265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.612254    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy"
Oct 30 22:32:04.612348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.612338    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy"
Oct 30 22:32:04.612476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.612466    4581 cpu_manager.go:414] "containerMap: removing container" podUID="af752c40-842e-4c32-990f-da63c2a0066f" containerName="payload-pre-processing"
Oct 30 22:32:04.612549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.612540    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af752c40-842e-4c32-990f-da63c2a0066f" containerName="payload-pre-processing"
Oct 30 22:32:04.612861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.612848    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="af752c40-842e-4c32-990f-da63c2a0066f" containerName="payload-pre-processing"
Oct 30 22:32:04.612966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.612957    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d3182d-81a5-4591-9a6d-ce1bfffeb438" containerName="istio-proxy"
Oct 30 22:32:04.613042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.613035    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5467bf2c-3a86-4dc1-9975-381cbb821574" containerName="istio-proxy"
Oct 30 22:32:04.781244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.777068    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"]
Oct 30 22:32:04.810245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.804371    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:04.822266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.819356    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4f7439\""
Oct 30 22:32:04.822266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.819489    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4f7439-dockercfg-lnz2d\""
Oct 30 22:32:04.981971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.981662    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrsf\" (UniqueName: \"kubernetes.io/projected/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-kube-api-access-zvrsf\") pod \"payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk\" (UID: \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:04.981971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:04.981834    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk\" (UID: \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:05.130243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.082934    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrsf\" (UniqueName: \"kubernetes.io/projected/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-kube-api-access-zvrsf\") pod \"payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk\" (UID: \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:05.155128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.130627    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk\" (UID: \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:05.155943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.131803    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk\" (UID: \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:05.156138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.114066    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"]
Oct 30 22:32:05.174187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.146401    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrsf\" (UniqueName: \"kubernetes.io/projected/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-kube-api-access-zvrsf\") pod \"payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk\" (UID: \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:05.247836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.247802    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"]
Oct 30 22:32:05.248045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.247948    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:05.358656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.358614    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8\" (UID: \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:05.360152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.360124    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdhfj\" (UniqueName: \"kubernetes.io/projected/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-kube-api-access-xdhfj\") pod \"payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8\" (UID: \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:05.373294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.373259    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:05.465296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.462883    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdhfj\" (UniqueName: \"kubernetes.io/projected/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-kube-api-access-xdhfj\") pod \"payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8\" (UID: \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:05.465296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.463258    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8\" (UID: \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:05.486822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.486783    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8\" (UID: \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:05.489711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.489327    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdhfj\" (UniqueName: \"kubernetes.io/projected/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-kube-api-access-xdhfj\") pod \"payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8\" (UID: \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:05.623950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.623677    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:05.820717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:05.820190    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"]
Oct 30 22:32:06.108038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.108007    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"]
Oct 30 22:32:06.388705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.388673    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8" event={"ID":"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61","Type":"ContainerStarted","Data":"cb43fd2cc3adb9ec5e4411594930f55b8678d693a90d5b333d27bf92cf3607bc"}
Oct 30 22:32:06.389768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.389744    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8" event={"ID":"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61","Type":"ContainerStarted","Data":"fc4dbdcca83485c42f34723b66d18003147777872aaf716b6f114c787565e9cd"}
Oct 30 22:32:06.389928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.389917    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:06.392051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.392030    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk" event={"ID":"4bdfa881-7aec-4cec-81cc-e15c0efc41a2","Type":"ContainerStarted","Data":"bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379"}
Oct 30 22:32:06.392244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.392201    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk" event={"ID":"4bdfa881-7aec-4cec-81cc-e15c0efc41a2","Type":"ContainerStarted","Data":"e472a840b917abc25ffae1165653f38f38425b0309401871c3daee4e0e1c4ee1"}
Oct 30 22:32:06.392414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.392398    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:06.430319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.430258    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8" podStartSLOduration=2.430239029 podStartE2EDuration="2.430239029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:06.422656465 +0000 UTC m=+1510.461595350" watchObservedRunningTime="2026-10-30 22:32:06.430239029 +0000 UTC m=+1510.469177912"
Oct 30 22:32:06.488133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:06.488069    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk" podStartSLOduration=2.48805174 podStartE2EDuration="2.48805174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:06.460625583 +0000 UTC m=+1510.499564471" watchObservedRunningTime="2026-10-30 22:32:06.48805174 +0000 UTC m=+1510.526990626"
Oct 30 22:32:08.122733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:08.122688    4581 patch_prober.go:29] interesting pod/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": read tcp 10.133.0.2:48960->10.133.0.182:15021: read: connection reset by peer" start-of-body=
Oct 30 22:32:08.123189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:08.122753    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": read tcp 10.133.0.2:48960->10.133.0.182:15021: read: connection reset by peer"
Oct 30 22:32:08.416129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:08.416041    4581 generic.go:356] "Generic (PLEG): container finished" podID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerID="9d9f02ca54d4c40c8b19ef4d06279b7cc8dd9b53c890b17c0f3aa19c6d09cb55" exitCode=0
Oct 30 22:32:08.416342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:08.416127    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" event={"ID":"6511e97e-31ed-4652-b7a1-e0661ea7c88e","Type":"ContainerDied","Data":"9d9f02ca54d4c40c8b19ef4d06279b7cc8dd9b53c890b17c0f3aa19c6d09cb55"}
Oct 30 22:32:08.416721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:08.416703    4581 scope.go:122] "RemoveContainer" containerID="9d9f02ca54d4c40c8b19ef4d06279b7cc8dd9b53c890b17c0f3aa19c6d09cb55"
Oct 30 22:32:08.418563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:08.418536    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:08.418630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:08.418607    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:08.418668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:08.418644    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:09.429733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.429631    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" event={"ID":"6511e97e-31ed-4652-b7a1-e0661ea7c88e","Type":"ContainerStarted","Data":"c71979a8e7771f8501175b03430e90e8cab573792dc43ee980f7a42dbb0162ef"}
Oct 30 22:32:09.814331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.814200    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"]
Oct 30 22:32:09.819674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.819645    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:09.833741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.832436    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:32:09.833741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.833069    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:32:09.838770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.838743    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"]
Oct 30 22:32:09.840484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.840459    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:32:09.840484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.840464    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:32:09.900672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.900502    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:09.900672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.900547    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffkw\" (UniqueName: \"kubernetes.io/projected/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-kube-api-access-pffkw\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:09.900931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:09.900711    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:10.002274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.002186    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:10.002274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.002281    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffkw\" (UniqueName: \"kubernetes.io/projected/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-kube-api-access-pffkw\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:10.002536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.002498    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:10.003048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.003023    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:10.005455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.005425    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:10.019719 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.019685    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffkw\" (UniqueName: \"kubernetes.io/projected/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-kube-api-access-pffkw\") pod \"maas-api-revoke-keys-e2e-api-a-370ec2-jz629\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:10.145139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.145087    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:10.197529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.197499    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:32:10.197902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.197877    4581 patch_prober.go:29] interesting pod/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:32:10.198650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.198615    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 22:32:10.541771 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:32:10.541740    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608392f7_a5a1_47a8_ba9d_6b84f8b0bc7b.slice/crio-757fa90478bb66e40c1f8921a84eeaccfce042ab323fd0259189cd28467466eb WatchSource:0}: Error finding container 757fa90478bb66e40c1f8921a84eeaccfce042ab323fd0259189cd28467466eb: Status 404 returned error can't find the container with id 757fa90478bb66e40c1f8921a84eeaccfce042ab323fd0259189cd28467466eb
Oct 30 22:32:10.542672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:10.542650    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"]
Oct 30 22:32:10.824379 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:32:10.824327    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:32:11.197424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:11.197387    4581 patch_prober.go:29] interesting pod/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:32:11.197684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:11.197660    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 22:32:11.503271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:11.500414    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629" event={"ID":"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b","Type":"ContainerStarted","Data":"b0555a09fbb00682bd65e41d19e8cf82f943c1a710538801a2c475fd4e7b8140"}
Oct 30 22:32:11.503271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:11.500485    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629" event={"ID":"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b","Type":"ContainerStarted","Data":"757fa90478bb66e40c1f8921a84eeaccfce042ab323fd0259189cd28467466eb"}
Oct 30 22:32:11.535689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:11.535626    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629" podStartSLOduration=2.535606305 podStartE2EDuration="2.535606305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:11.53352879 +0000 UTC m=+1515.572467679" watchObservedRunningTime="2026-10-30 22:32:11.535606305 +0000 UTC m=+1515.574545193"
Oct 30 22:32:12.198247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:12.197820    4581 patch_prober.go:29] interesting pod/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:32:12.198247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:12.197879    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 22:32:12.515954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:12.515867    4581 generic.go:356] "Generic (PLEG): container finished" podID="608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b" containerID="b0555a09fbb00682bd65e41d19e8cf82f943c1a710538801a2c475fd4e7b8140" exitCode=0
Oct 30 22:32:12.516108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:12.515970    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629" event={"ID":"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b","Type":"ContainerDied","Data":"b0555a09fbb00682bd65e41d19e8cf82f943c1a710538801a2c475fd4e7b8140"}
Oct 30 22:32:13.197235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:13.197175    4581 patch_prober.go:29] interesting pod/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:32:13.197453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:13.197254    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 22:32:14.013291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.013251    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:14.189321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.165142    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-xks-ca\") pod \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") "
Oct 30 22:32:14.189954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.189922    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-kube-api-access-pffkw\" (UniqueName: \"kubernetes.io/projected/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-kube-api-access-pffkw\") pod \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") "
Oct 30 22:32:14.190315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.190296    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-openshift-ca\") pod \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\" (UID: \"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b\") "
Oct 30 22:32:14.191638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.191584    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-openshift-ca" pod "608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b" (UID: "608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:14.208696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.207937    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-kube-api-access-pffkw" pod "608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b" (UID: "608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b"). InnerVolumeSpecName "kube-api-access-pffkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:14.208696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.208130    4581 patch_prober.go:29] interesting pod/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:32:14.208696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.208176    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 22:32:14.208696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.208652    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-xks-ca" pod "608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b" (UID: "608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:14.292937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.292899    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:14.292937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.292930    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:14.292937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.292940    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pffkw\" (UniqueName: \"kubernetes.io/projected/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b-kube-api-access-pffkw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:14.606967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.606933    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629" event={"ID":"608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b","Type":"ContainerDied","Data":"757fa90478bb66e40c1f8921a84eeaccfce042ab323fd0259189cd28467466eb"}
Oct 30 22:32:14.607350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.607332    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="757fa90478bb66e40c1f8921a84eeaccfce042ab323fd0259189cd28467466eb"
Oct 30 22:32:14.607473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:14.606992    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"
Oct 30 22:32:15.197721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:15.197679    4581 patch_prober.go:29] interesting pod/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:32:15.198296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:15.197757    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 22:32:15.927380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:15.927344    4581 patch_prober.go:29] interesting pod/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:32:15.927559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:15.927447    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:32:16.254410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.251684    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:32:16.272590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.263663    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:32:16.272590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.272506    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:32:16.628380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.628343    4581 generic.go:356] "Generic (PLEG): container finished" podID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerID="cdeac9cb3e79b8a74f179f3761b0a685825c6a5d4f00418a2d5ad6676e8e9648" exitCode=0
Oct 30 22:32:16.628559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.628419    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" event={"ID":"dfb3f78a-618e-484e-b9ba-49ee3dd6c589","Type":"ContainerDied","Data":"cdeac9cb3e79b8a74f179f3761b0a685825c6a5d4f00418a2d5ad6676e8e9648"}
Oct 30 22:32:16.628902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.628889    4581 scope.go:122] "RemoveContainer" containerID="cdeac9cb3e79b8a74f179f3761b0a685825c6a5d4f00418a2d5ad6676e8e9648"
Oct 30 22:32:16.630957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.630927    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:16.631064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.630995    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:16.631064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:16.631027    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:17.409480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:17.409443    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:32:17.640800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:17.640765    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" event={"ID":"dfb3f78a-618e-484e-b9ba-49ee3dd6c589","Type":"ContainerStarted","Data":"5548a25208e2e07fa81676dd8ceb0eaa6acc9dcfaf07afd74906300dcac66571"}
Oct 30 22:32:18.254723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:18.254685    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:32:18.255068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:18.255043    4581 patch_prober.go:29] interesting pod/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:32:18.255309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:18.255278    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 22:32:18.419939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:18.419905    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:32:19.250359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:19.250321    4581 patch_prober.go:29] interesting pod/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:32:19.250674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:19.250641    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 22:32:20.250208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:20.250173    4581 patch_prober.go:29] interesting pod/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:32:20.250903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:20.250309    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 22:32:21.250449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:21.250410    4581 patch_prober.go:29] interesting pod/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:32:21.251129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:21.251093    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 22:32:22.276899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:22.276848    4581 patch_prober.go:29] interesting pod/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:32:22.277516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:22.277483    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 22:32:23.040344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:23.040300    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"]
Oct 30 22:32:23.063467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:23.050532    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b0ca83-vs2dl"]
Oct 30 22:32:23.250444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:23.249991    4581 patch_prober.go:29] interesting pod/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:32:23.250444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:23.250045    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 22:32:24.105617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.105579    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072910b5-0de0-4afd-a797-73f83f6afd51" path="/var/lib/kubelet/pods/072910b5-0de0-4afd-a797-73f83f6afd51/volumes"
Oct 30 22:32:24.120967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.120932    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"]
Oct 30 22:32:24.141985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.141944    4581 cpu_manager.go:414] "containerMap: removing container" podUID="608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b" containerName="revoke-keys"
Oct 30 22:32:24.141985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.141982    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b" containerName="revoke-keys"
Oct 30 22:32:24.142262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.142254    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b" containerName="revoke-keys"
Oct 30 22:32:24.189784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.188316    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.190152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.190128    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"]
Oct 30 22:32:24.204653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.204626    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-62236a-openshift-default-dockercfg-dfsdp\""
Oct 30 22:32:24.457296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.443661    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:32:24.501243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.478091    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:32:24.501243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.478389    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-certs\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.501243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.478882    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxn5j\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-kube-api-access-lxn5j\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.501243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.479115    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-credential-socket\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.501243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.479813    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-data\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.501243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.480274    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-envoy\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.501243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.480478    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istiod-ca-cert\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.501243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.480726    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-token\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.523977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.509193    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-podinfo\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.523977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.509334    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-socket\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.610648    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612146    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-envoy\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612190    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istiod-ca-cert\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612271    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-token\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612403    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-podinfo\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612489    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-socket\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612571    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-certs\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612671    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxn5j\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-kube-api-access-lxn5j\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612717    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-credential-socket\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.612828    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-data\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.613359    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-data\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.614575    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-socket\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.624430    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-certs\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.626241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.625548    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-credential-socket\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.629770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.629744    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istiod-ca-cert\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.632050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.632025    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-envoy\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.645807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.645780    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-token\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.647063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.647028    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-podinfo\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.647824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.647800    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxn5j\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-kube-api-access-lxn5j\") pod \"e2e-api-b-62236a-openshift-default-899986fcf-8kdtl\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") " pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:24.741647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:24.741551    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:25.142836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:25.142810    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"]
Oct 30 22:32:25.143394 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:32:25.143368    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb519829a_bcbe_4c2e_a4ca_28b4bd299dd4.slice/crio-a3a513099ee3b18c83af7d93258d77f948c13b3343d07d107f3456769845b737 WatchSource:0}: Error finding container a3a513099ee3b18c83af7d93258d77f948c13b3343d07d107f3456769845b737: Status 404 returned error can't find the container with id a3a513099ee3b18c83af7d93258d77f948c13b3343d07d107f3456769845b737
Oct 30 22:32:25.146680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:25.146643    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:25.146780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:25.146715    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:25.146780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:25.146745    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:26.077128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:26.077077    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" event={"ID":"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4","Type":"ContainerStarted","Data":"7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8"}
Oct 30 22:32:26.077388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:26.077193    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" event={"ID":"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4","Type":"ContainerStarted","Data":"a3a513099ee3b18c83af7d93258d77f948c13b3343d07d107f3456769845b737"}
Oct 30 22:32:26.137286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:26.137234    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podStartSLOduration=2.137203995 podStartE2EDuration="2.137203995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:26.126086729 +0000 UTC m=+1530.165025627" watchObservedRunningTime="2026-10-30 22:32:26.137203995 +0000 UTC m=+1530.176142881"
Oct 30 22:32:26.742960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:26.742924    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:26.941679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:26.941640    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:27.091340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:27.091246    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:27.092444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:27.092422    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:32:32.595957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.595919    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"]
Oct 30 22:32:32.600069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.600040    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.617385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.617355    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"]
Oct 30 22:32:32.617598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.617582    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:32:32.620747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.620721    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:32:32.624334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.624307    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:32:32.624469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.624385    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:32:32.697156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.697111    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.697757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.697709    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.698096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.698075    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mg6\" (UniqueName: \"kubernetes.io/projected/1bad6af9-d379-4152-9fd1-2679605e6cc3-kube-api-access-m4mg6\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.803572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.803535    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mg6\" (UniqueName: \"kubernetes.io/projected/1bad6af9-d379-4152-9fd1-2679605e6cc3-kube-api-access-m4mg6\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.803980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.803952    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.804614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.804262    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.804802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.804777    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.811239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.810647    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.823712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.823684    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mg6\" (UniqueName: \"kubernetes.io/projected/1bad6af9-d379-4152-9fd1-2679605e6cc3-kube-api-access-m4mg6\") pod \"maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:32.918100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:32.918057    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:33.322355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:33.322314    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"]
Oct 30 22:32:33.324277 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:32:33.322807    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bad6af9_d379_4152_9fd1_2679605e6cc3.slice/crio-4113a518db1ed08758692b2849d804a27d7c1a0327b27ee193f74ca1010fafdb WatchSource:0}: Error finding container 4113a518db1ed08758692b2849d804a27d7c1a0327b27ee193f74ca1010fafdb: Status 404 returned error can't find the container with id 4113a518db1ed08758692b2849d804a27d7c1a0327b27ee193f74ca1010fafdb
Oct 30 22:32:33.609268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:33.607910    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw" event={"ID":"1bad6af9-d379-4152-9fd1-2679605e6cc3","Type":"ContainerStarted","Data":"4113a518db1ed08758692b2849d804a27d7c1a0327b27ee193f74ca1010fafdb"}
Oct 30 22:32:34.621705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:34.621673    4581 generic.go:356] "Generic (PLEG): container finished" podID="1bad6af9-d379-4152-9fd1-2679605e6cc3" containerID="3a2bc800b56baad9dd41113ab30cac48244094b1871d2c8e221e2e394dd2f05e" exitCode=0
Oct 30 22:32:34.622093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:34.621756    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw" event={"ID":"1bad6af9-d379-4152-9fd1-2679605e6cc3","Type":"ContainerDied","Data":"3a2bc800b56baad9dd41113ab30cac48244094b1871d2c8e221e2e394dd2f05e"}
Oct 30 22:32:35.580247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.577815    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"]
Oct 30 22:32:35.606250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.588181    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:35.644241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.641509    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6922d3\""
Oct 30 22:32:35.736296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.678708    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"]
Oct 30 22:32:35.736296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.706242    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6922d3-dockercfg-v5fn8\""
Oct 30 22:32:35.756259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.739950    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725481-e08a-4302-a783-efd11a4a4338-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m\" (UID: \"08725481-e08a-4302-a783-efd11a4a4338\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:35.756259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.740013    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/08725481-e08a-4302-a783-efd11a4a4338-kube-api-access-hh84x\") pod \"payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m\" (UID: \"08725481-e08a-4302-a783-efd11a4a4338\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:35.876241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.864565    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725481-e08a-4302-a783-efd11a4a4338-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m\" (UID: \"08725481-e08a-4302-a783-efd11a4a4338\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:35.876241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.864622    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/08725481-e08a-4302-a783-efd11a4a4338-kube-api-access-hh84x\") pod \"payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m\" (UID: \"08725481-e08a-4302-a783-efd11a4a4338\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:35.876241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.866085    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725481-e08a-4302-a783-efd11a4a4338-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m\" (UID: \"08725481-e08a-4302-a783-efd11a4a4338\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:35.929204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.929167    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/08725481-e08a-4302-a783-efd11a4a4338-kube-api-access-hh84x\") pod \"payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m\" (UID: \"08725481-e08a-4302-a783-efd11a4a4338\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:35.943542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:35.943511    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:36.212553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.212525    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:36.298198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.298088    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-xks-ca\") pod \"1bad6af9-d379-4152-9fd1-2679605e6cc3\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") "
Oct 30 22:32:36.298415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.298324    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bad6af9-d379-4152-9fd1-2679605e6cc3-kube-api-access-m4mg6\" (UniqueName: \"kubernetes.io/projected/1bad6af9-d379-4152-9fd1-2679605e6cc3-kube-api-access-m4mg6\") pod \"1bad6af9-d379-4152-9fd1-2679605e6cc3\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") "
Oct 30 22:32:36.298573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.298524    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-openshift-ca\") pod \"1bad6af9-d379-4152-9fd1-2679605e6cc3\" (UID: \"1bad6af9-d379-4152-9fd1-2679605e6cc3\") "
Oct 30 22:32:36.299870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.299421    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-openshift-ca" pod "1bad6af9-d379-4152-9fd1-2679605e6cc3" (UID: "1bad6af9-d379-4152-9fd1-2679605e6cc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:36.299870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.299576    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:36.301268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.301242    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-xks-ca" pod "1bad6af9-d379-4152-9fd1-2679605e6cc3" (UID: "1bad6af9-d379-4152-9fd1-2679605e6cc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:36.301818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.301791    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bad6af9-d379-4152-9fd1-2679605e6cc3-kube-api-access-m4mg6" pod "1bad6af9-d379-4152-9fd1-2679605e6cc3" (UID: "1bad6af9-d379-4152-9fd1-2679605e6cc3"). InnerVolumeSpecName "kube-api-access-m4mg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:36.347102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.347064    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"]
Oct 30 22:32:36.351807 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:32:36.351765    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08725481_e08a_4302_a783_efd11a4a4338.slice/crio-9eef16a0182eb263a5eba666514cc21a9b69807e580d2004b601d8f7c00a740d WatchSource:0}: Error finding container 9eef16a0182eb263a5eba666514cc21a9b69807e580d2004b601d8f7c00a740d: Status 404 returned error can't find the container with id 9eef16a0182eb263a5eba666514cc21a9b69807e580d2004b601d8f7c00a740d
Oct 30 22:32:36.401123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.401089    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad6af9-d379-4152-9fd1-2679605e6cc3-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:36.401322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.401129    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mg6\" (UniqueName: \"kubernetes.io/projected/1bad6af9-d379-4152-9fd1-2679605e6cc3-kube-api-access-m4mg6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:36.955494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.955461    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"
Oct 30 22:32:36.955494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.955483    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw" event={"ID":"1bad6af9-d379-4152-9fd1-2679605e6cc3","Type":"ContainerDied","Data":"4113a518db1ed08758692b2849d804a27d7c1a0327b27ee193f74ca1010fafdb"}
Oct 30 22:32:36.956050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.955544    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4113a518db1ed08758692b2849d804a27d7c1a0327b27ee193f74ca1010fafdb"
Oct 30 22:32:36.958409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.958366    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m" event={"ID":"08725481-e08a-4302-a783-efd11a4a4338","Type":"ContainerStarted","Data":"7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b"}
Oct 30 22:32:36.958689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.958665    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m" event={"ID":"08725481-e08a-4302-a783-efd11a4a4338","Type":"ContainerStarted","Data":"9eef16a0182eb263a5eba666514cc21a9b69807e580d2004b601d8f7c00a740d"}
Oct 30 22:32:36.958881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.958865    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:36.992382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:36.992321    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m" podStartSLOduration=1.9923007209999999 podStartE2EDuration="1.992300721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:36.99137 +0000 UTC m=+1541.030308887" watchObservedRunningTime="2026-10-30 22:32:36.992300721 +0000 UTC m=+1541.031239608"
Oct 30 22:32:39.925164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:39.925121    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"]
Oct 30 22:32:39.925670 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:39.925423    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy" containerID="cri-o://a0ff16d26caf5351a73c1606c1ed702291ea5ff90a719b638efdd00092954663" gracePeriod=30
Oct 30 22:32:40.046473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:40.046411    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"]
Oct 30 22:32:40.052353 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:40.052320    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6935ede8-9cx5m"]
Oct 30 22:32:40.103110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:40.103063    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9" path="/var/lib/kubelet/pods/8df9bd00-b4c0-4ecc-bb2d-818d3fd1eea9/volumes"
Oct 30 22:32:40.822244 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:32:40.822195    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:32:41.007927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:41.007896    4581 generic.go:356] "Generic (PLEG): container finished" podID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerID="1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8" exitCode=0
Oct 30 22:32:41.008367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:41.007970    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" event={"ID":"ae4b4443-3f69-43a0-9a55-118b93e68702","Type":"ContainerDied","Data":"1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8"}
Oct 30 22:32:41.008501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:41.008485    4581 scope.go:122] "RemoveContainer" containerID="1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8"
Oct 30 22:32:41.010559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:41.010533    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:41.010650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:41.010600    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:41.010650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:41.010642    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:32:42.019284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:42.019211    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" event={"ID":"ae4b4443-3f69-43a0-9a55-118b93e68702","Type":"ContainerStarted","Data":"b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a"}
Oct 30 22:32:42.875238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:42.873611    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:32:42.882881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:42.882355    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:32:42.882881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:42.882424    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 22:32:43.193745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:43.193647    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"]
Oct 30 22:32:43.221640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:43.221605    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"]
Oct 30 22:32:43.871671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:43.869844    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:32:43.871671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:43.869904    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 22:32:44.887282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:44.881482    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:32:44.887282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:44.881549    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 22:32:45.096190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.096144    4581 generic.go:356] "Generic (PLEG): container finished" podID="16c79349-2330-434d-8c58-39da2ad67c03" containerID="a0ff16d26caf5351a73c1606c1ed702291ea5ff90a719b638efdd00092954663" exitCode=0
Oct 30 22:32:45.096516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.096496    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" event={"ID":"16c79349-2330-434d-8c58-39da2ad67c03","Type":"ContainerDied","Data":"a0ff16d26caf5351a73c1606c1ed702291ea5ff90a719b638efdd00092954663"}
Oct 30 22:32:45.096648 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.096636    4581 scope.go:122] "RemoveContainer" containerID="f63623688584991420088798e71d800a8d2fab79b5b37d9724a008bbdb9a84be"
Oct 30 22:32:45.328023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.327992    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:32:45.431867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.431770    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-kube-api-access-xbxb2\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-kube-api-access-xbxb2\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.431886    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-data\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.431988    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-socket\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432169    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16c79349-2330-434d-8c58-39da2ad67c03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c79349-2330-434d-8c58-39da2ad67c03-istio-podinfo\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432184    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-data" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.432352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432305    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-socket" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.432352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432325    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c79349-2330-434d-8c58-39da2ad67c03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c79349-2330-434d-8c58-39da2ad67c03-istiod-ca-cert\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432413    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-credential-socket\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432461    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-envoy\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432501    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-istio-token\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-istio-token\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432618    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-certs\") pod \"16c79349-2330-434d-8c58-39da2ad67c03\" (UID: \"16c79349-2330-434d-8c58-39da2ad67c03\") "
Oct 30 22:32:45.432734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432708    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-credential-socket" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.432824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.432752    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c79349-2330-434d-8c58-39da2ad67c03-istiod-ca-cert" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:45.433063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.433037    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-certs" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.433691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.433644    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c79349-2330-434d-8c58-39da2ad67c03-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.433691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.433677    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.433691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.433692    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.433903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.433707    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.433903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.433720    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.437679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.437645    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16c79349-2330-434d-8c58-39da2ad67c03-istio-podinfo" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:45.437767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.437691    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-kube-api-access-xbxb2" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "kube-api-access-xbxb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:45.437959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.437877    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-istio-token" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:45.437959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.437874    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-envoy" pod "16c79349-2330-434d-8c58-39da2ad67c03" (UID: "16c79349-2330-434d-8c58-39da2ad67c03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.535177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.535141    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbxb2\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-kube-api-access-xbxb2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.535177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.535170    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c79349-2330-434d-8c58-39da2ad67c03-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.535177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.535180    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c79349-2330-434d-8c58-39da2ad67c03-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.535470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.535194    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c79349-2330-434d-8c58-39da2ad67c03-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:45.870877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.868613    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:32:45.870877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:45.868690    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 22:32:46.119916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:46.119883    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"
Oct 30 22:32:46.120422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:46.119945    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5" event={"ID":"16c79349-2330-434d-8c58-39da2ad67c03","Type":"ContainerDied","Data":"b704f42538c84b542e857a2c89b496552357048674aece1895b4e3b673088ce1"}
Oct 30 22:32:46.120422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:46.120005    4581 scope.go:122] "RemoveContainer" containerID="a0ff16d26caf5351a73c1606c1ed702291ea5ff90a719b638efdd00092954663"
Oct 30 22:32:46.199324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:46.199201    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"]
Oct 30 22:32:46.206624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:46.206589    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-k4nm5"]
Oct 30 22:32:46.871743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:46.871693    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:32:46.872059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:46.872020    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 22:32:47.869242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:47.868652    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:32:47.869242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:47.868717    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 22:32:47.972760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:47.972718    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:32:48.109310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.109268    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c79349-2330-434d-8c58-39da2ad67c03" path="/var/lib/kubelet/pods/16c79349-2330-434d-8c58-39da2ad67c03/volumes"
Oct 30 22:32:48.205349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.202396    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm" podUID="ecb7fc7d-8e84-4012-8ed1-28b1516d99ad" containerName="payload-processing" containerID="cri-o://c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b" gracePeriod=25
Oct 30 22:32:48.222528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.222490    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5" podUID="02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14" containerName="payload-pre-processing" containerID="cri-o://01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e" gracePeriod=25
Oct 30 22:32:48.671805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.671772    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:32:48.730480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.730446    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:32:48.823914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.823287    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-kube-api-access-vfg54\" (UniqueName: \"kubernetes.io/projected/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-kube-api-access-vfg54\") pod \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\" (UID: \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\") "
Oct 30 22:32:48.824118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.824076    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-plugins-config-volume\") pod \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\" (UID: \"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad\") "
Oct 30 22:32:48.831355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.831309    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-plugins-config-volume" pod "ecb7fc7d-8e84-4012-8ed1-28b1516d99ad" (UID: "ecb7fc7d-8e84-4012-8ed1-28b1516d99ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:48.842612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.841877    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-kube-api-access-vfg54" pod "ecb7fc7d-8e84-4012-8ed1-28b1516d99ad" (UID: "ecb7fc7d-8e84-4012-8ed1-28b1516d99ad"). InnerVolumeSpecName "kube-api-access-vfg54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:48.874531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.874436    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:32:48.875073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.875052    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:32:48.875999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.875976    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 22:32:48.927000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.926971    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-plugins-config-volume\") pod \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\" (UID: \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\") "
Oct 30 22:32:48.927419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.927394    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-plugins-config-volume" pod "02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14" (UID: "02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:48.927536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.927432    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-kube-api-access-qsxhs\" (UniqueName: \"kubernetes.io/projected/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-kube-api-access-qsxhs\") pod \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\" (UID: \"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14\") "
Oct 30 22:32:48.927962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.927938    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:48.927962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.927959    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:48.928137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.927971    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfg54\" (UniqueName: \"kubernetes.io/projected/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad-kube-api-access-vfg54\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:48.929811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:48.929786    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-kube-api-access-qsxhs" pod "02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14" (UID: "02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14"). InnerVolumeSpecName "kube-api-access-qsxhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:49.029229 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.029182    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxhs\" (UniqueName: \"kubernetes.io/projected/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14-kube-api-access-qsxhs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:32:49.191116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.190994    4581 generic.go:356] "Generic (PLEG): container finished" podID="02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14" containerID="01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e" exitCode=0
Oct 30 22:32:49.191342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.191240    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5" event={"ID":"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14","Type":"ContainerDied","Data":"01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e"}
Oct 30 22:32:49.191342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.191308    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5" event={"ID":"02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14","Type":"ContainerDied","Data":"8b55cc52df10178b202b2d9fe241dfb2be24b66ffe69bae00c76f5d036b0e006"}
Oct 30 22:32:49.191576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.191366    4581 scope.go:122] "RemoveContainer" containerID="01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e"
Oct 30 22:32:49.191576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.191562    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"
Oct 30 22:32:49.218574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.218540    4581 generic.go:356] "Generic (PLEG): container finished" podID="ecb7fc7d-8e84-4012-8ed1-28b1516d99ad" containerID="c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b" exitCode=0
Oct 30 22:32:49.219890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.219856    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"
Oct 30 22:32:49.233002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.221606    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm" event={"ID":"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad","Type":"ContainerDied","Data":"c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b"}
Oct 30 22:32:49.233002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.221663    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm" event={"ID":"ecb7fc7d-8e84-4012-8ed1-28b1516d99ad","Type":"ContainerDied","Data":"1346485ba9d9f7df71dc2b4b93b9d0265a6df091427e2703e4560598f956c811"}
Oct 30 22:32:49.260366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.260340    4581 scope.go:122] "RemoveContainer" containerID="01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e"
Oct 30 22:32:49.260719 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:32:49.260695    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e\": container with ID starting with 01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e not found: ID does not exist" containerID="01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e"
Oct 30 22:32:49.260784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.260732    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e"} err="failed to get container status \"01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e\": rpc error: code = NotFound desc = could not find container \"01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e\": container with ID starting with 01c216704cb75008602fac035142ce083a949b9b09f959598d37b746fa24b02e not found: ID does not exist"
Oct 30 22:32:49.260784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.260757    4581 scope.go:122] "RemoveContainer" containerID="c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b"
Oct 30 22:32:49.272100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.272078    4581 scope.go:122] "RemoveContainer" containerID="c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b"
Oct 30 22:32:49.272448 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:32:49.272421    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b\": container with ID starting with c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b not found: ID does not exist" containerID="c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b"
Oct 30 22:32:49.272525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.272459    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b"} err="failed to get container status \"c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b\": rpc error: code = NotFound desc = could not find container \"c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b\": container with ID starting with c64a122dd5cebaf610b6f1c791b7deadfaada66b1aa94e7bd39a63bd9942500b not found: ID does not exist"
Oct 30 22:32:49.290850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.290818    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"]
Oct 30 22:32:49.296064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.296036    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f443d6-6bcbc5c45f-72rp5"]
Oct 30 22:32:49.322938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.322912    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"]
Oct 30 22:32:49.329139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:49.329114    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f443d6-784b88c549-cwkhm"]
Oct 30 22:32:50.100526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:50.100493    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14" path="/var/lib/kubelet/pods/02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14/volumes"
Oct 30 22:32:50.101205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:50.101183    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb7fc7d-8e84-4012-8ed1-28b1516d99ad" path="/var/lib/kubelet/pods/ecb7fc7d-8e84-4012-8ed1-28b1516d99ad/volumes"
Oct 30 22:32:59.053190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:59.053147    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:32:59.054234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:59.054183    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:32:59.317405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:59.306311    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:32:59.497246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:59.494302    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:32:59.579300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:59.558423    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:32:59.579300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:59.572043    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:32:59.628246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:59.626669    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:32:59.814244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:32:59.809412    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:08.712633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.712551    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"]
Oct 30 22:33:08.713388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713369    4581 cpu_manager.go:414] "containerMap: removing container" podUID="02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14" containerName="payload-pre-processing"
Oct 30 22:33:08.713441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713389    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14" containerName="payload-pre-processing"
Oct 30 22:33:08.713441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713407    4581 cpu_manager.go:414] "containerMap: removing container" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy"
Oct 30 22:33:08.713441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713412    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy"
Oct 30 22:33:08.713441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713438    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1bad6af9-d379-4152-9fd1-2679605e6cc3" containerName="revoke-keys"
Oct 30 22:33:08.713576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713443    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bad6af9-d379-4152-9fd1-2679605e6cc3" containerName="revoke-keys"
Oct 30 22:33:08.713576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713465    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ecb7fc7d-8e84-4012-8ed1-28b1516d99ad" containerName="payload-processing"
Oct 30 22:33:08.713576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713470    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb7fc7d-8e84-4012-8ed1-28b1516d99ad" containerName="payload-processing"
Oct 30 22:33:08.713576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713568    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy"
Oct 30 22:33:08.713704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713582    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bad6af9-d379-4152-9fd1-2679605e6cc3" containerName="revoke-keys"
Oct 30 22:33:08.713704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713593    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb7fc7d-8e84-4012-8ed1-28b1516d99ad" containerName="payload-processing"
Oct 30 22:33:08.713704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.713602    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b2db7c-eaf7-4bcb-9075-e6ff0c2f9e14" containerName="payload-pre-processing"
Oct 30 22:33:08.719606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.719578    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:08.734393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.734346    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:33:08.742336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.741318    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:33:08.818659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.771631    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:08.818659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.771811    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:33:08.824809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.824778    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"]
Oct 30 22:33:08.924658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.924622    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:08.924863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.924760    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvb9v\" (UniqueName: \"kubernetes.io/projected/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-kube-api-access-mvb9v\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:08.924921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:08.924912    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:09.026011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.025902    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvb9v\" (UniqueName: \"kubernetes.io/projected/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-kube-api-access-mvb9v\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:09.026202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.026081    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:09.026350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.026290    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:09.026968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.026900    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:09.028807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.028781    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:09.042894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.042868    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvb9v\" (UniqueName: \"kubernetes.io/projected/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-kube-api-access-mvb9v\") pod \"maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:09.086891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.086851    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:09.406021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.405985    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"]
Oct 30 22:33:09.407761 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:33:09.407734    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915f0b7f_1740_4c1d_8bd7_4cbb7c7f7306.slice/crio-ee78ea83e9f9d79ecfb150997fa9e75923ba02bfc3c7bc98345d0bcc8321537d WatchSource:0}: Error finding container ee78ea83e9f9d79ecfb150997fa9e75923ba02bfc3c7bc98345d0bcc8321537d: Status 404 returned error can't find the container with id ee78ea83e9f9d79ecfb150997fa9e75923ba02bfc3c7bc98345d0bcc8321537d
Oct 30 22:33:09.838108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:09.838069    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7" event={"ID":"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306","Type":"ContainerStarted","Data":"ee78ea83e9f9d79ecfb150997fa9e75923ba02bfc3c7bc98345d0bcc8321537d"}
Oct 30 22:33:10.826413 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:33:10.826322    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:33:10.848569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:10.848533    4581 generic.go:356] "Generic (PLEG): container finished" podID="915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306" containerID="163202b471d2dacdf2ecbccbc2c30a0b7ed81508846fa1f872e46b7aab1dfc7e" exitCode=0
Oct 30 22:33:10.849014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:10.848636    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7" event={"ID":"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306","Type":"ContainerDied","Data":"163202b471d2dacdf2ecbccbc2c30a0b7ed81508846fa1f872e46b7aab1dfc7e"}
Oct 30 22:33:12.072411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.072378    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:12.179065 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.178963    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-xks-ca\") pod \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") "
Oct 30 22:33:12.179833 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.179805    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-kube-api-access-mvb9v\" (UniqueName: \"kubernetes.io/projected/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-kube-api-access-mvb9v\") pod \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") "
Oct 30 22:33:12.180654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.180631    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-openshift-ca\") pod \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\" (UID: \"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306\") "
Oct 30 22:33:12.181154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.181109    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-openshift-ca" pod "915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306" (UID: "915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:12.181901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.181863    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:12.188262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.185727    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-xks-ca" pod "915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306" (UID: "915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:12.188262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.185729    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-kube-api-access-mvb9v" pod "915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306" (UID: "915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306"). InnerVolumeSpecName "kube-api-access-mvb9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:12.284601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.284568    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:12.284868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.284839    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvb9v\" (UniqueName: \"kubernetes.io/projected/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306-kube-api-access-mvb9v\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:12.872925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.872898    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"
Oct 30 22:33:12.873132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.872944    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7" event={"ID":"915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306","Type":"ContainerDied","Data":"ee78ea83e9f9d79ecfb150997fa9e75923ba02bfc3c7bc98345d0bcc8321537d"}
Oct 30 22:33:12.873132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:12.873004    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee78ea83e9f9d79ecfb150997fa9e75923ba02bfc3c7bc98345d0bcc8321537d"
Oct 30 22:33:19.171132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:19.171090    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"]
Oct 30 22:33:24.178244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:24.177536    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z" podUID="3c9efcdb-4dbf-4736-951b-66a2baece0e1" containerName="payload-processing" containerID="cri-o://1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724" gracePeriod=25
Oct 30 22:33:24.559787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:24.559757    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:33:24.598991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:24.598958    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9efcdb-4dbf-4736-951b-66a2baece0e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9efcdb-4dbf-4736-951b-66a2baece0e1-plugins-config-volume\") pod \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\" (UID: \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\") "
Oct 30 22:33:24.599307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:24.599283    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9efcdb-4dbf-4736-951b-66a2baece0e1-kube-api-access-9cqr7\" (UniqueName: \"kubernetes.io/projected/3c9efcdb-4dbf-4736-951b-66a2baece0e1-kube-api-access-9cqr7\") pod \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\" (UID: \"3c9efcdb-4dbf-4736-951b-66a2baece0e1\") "
Oct 30 22:33:24.599634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:24.599397    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9efcdb-4dbf-4736-951b-66a2baece0e1-plugins-config-volume" pod "3c9efcdb-4dbf-4736-951b-66a2baece0e1" (UID: "3c9efcdb-4dbf-4736-951b-66a2baece0e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:24.600413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:24.600041    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9efcdb-4dbf-4736-951b-66a2baece0e1-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:24.605605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:24.603200    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9efcdb-4dbf-4736-951b-66a2baece0e1-kube-api-access-9cqr7" pod "3c9efcdb-4dbf-4736-951b-66a2baece0e1" (UID: "3c9efcdb-4dbf-4736-951b-66a2baece0e1"). InnerVolumeSpecName "kube-api-access-9cqr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:24.701649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:24.701614    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cqr7\" (UniqueName: \"kubernetes.io/projected/3c9efcdb-4dbf-4736-951b-66a2baece0e1-kube-api-access-9cqr7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:25.029349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.029312    4581 generic.go:356] "Generic (PLEG): container finished" podID="3c9efcdb-4dbf-4736-951b-66a2baece0e1" containerID="1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724" exitCode=0
Oct 30 22:33:25.029583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.029368    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"
Oct 30 22:33:25.029583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.029387    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z" event={"ID":"3c9efcdb-4dbf-4736-951b-66a2baece0e1","Type":"ContainerDied","Data":"1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724"}
Oct 30 22:33:25.029583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.029431    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z" event={"ID":"3c9efcdb-4dbf-4736-951b-66a2baece0e1","Type":"ContainerDied","Data":"209e77a495e978d0fbd0086406699efa41842fac3f9b30e453c9008d9459e244"}
Oct 30 22:33:25.029583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.029457    4581 scope.go:122] "RemoveContainer" containerID="1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724"
Oct 30 22:33:25.044385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.044360    4581 scope.go:122] "RemoveContainer" containerID="1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724"
Oct 30 22:33:25.044709 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:33:25.044689    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724\": container with ID starting with 1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724 not found: ID does not exist" containerID="1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724"
Oct 30 22:33:25.044763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.044720    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724"} err="failed to get container status \"1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724\": rpc error: code = NotFound desc = could not find container \"1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724\": container with ID starting with 1ef6d778ae6b9e2d9dfef9de6bbb3644af442f200a2c93fe93334f41d83cf724 not found: ID does not exist"
Oct 30 22:33:25.076823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.076790    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"]
Oct 30 22:33:25.086163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:25.086128    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-de31dd-7d9cf5d4f-9lr2z"]
Oct 30 22:33:26.101077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.101043    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9efcdb-4dbf-4736-951b-66a2baece0e1" path="/var/lib/kubelet/pods/3c9efcdb-4dbf-4736-951b-66a2baece0e1/volumes"
Oct 30 22:33:26.603402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.603368    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"]
Oct 30 22:33:26.604420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604400    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3c9efcdb-4dbf-4736-951b-66a2baece0e1" containerName="payload-processing"
Oct 30 22:33:26.604486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604423    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9efcdb-4dbf-4736-951b-66a2baece0e1" containerName="payload-processing"
Oct 30 22:33:26.604486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604461    4581 cpu_manager.go:414] "containerMap: removing container" podUID="915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306" containerName="revoke-keys"
Oct 30 22:33:26.604486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604466    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306" containerName="revoke-keys"
Oct 30 22:33:26.604588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604487    4581 cpu_manager.go:414] "containerMap: removing container" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy"
Oct 30 22:33:26.604588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604492    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy"
Oct 30 22:33:26.604652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604646    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9efcdb-4dbf-4736-951b-66a2baece0e1" containerName="payload-processing"
Oct 30 22:33:26.604694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604661    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306" containerName="revoke-keys"
Oct 30 22:33:26.604694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.604670    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c79349-2330-434d-8c58-39da2ad67c03" containerName="istio-proxy"
Oct 30 22:33:26.609754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.609734    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.622463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.622436    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:26.627575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.627553    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"]
Oct 30 22:33:26.630444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.630419    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:33:26.630566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.630440    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:33:26.632034 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.632011    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:33:26.659599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.659528    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.660274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.660191    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.660400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.660341    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kq7\" (UniqueName: \"kubernetes.io/projected/7a720020-91b0-4d87-97f5-c8418dd80cd3-kube-api-access-n7kq7\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.762644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.762609    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.762949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.762929    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.763018 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.762970    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kq7\" (UniqueName: \"kubernetes.io/projected/7a720020-91b0-4d87-97f5-c8418dd80cd3-kube-api-access-n7kq7\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.763694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.763672    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.765351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.765329    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.781070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.781036    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kq7\" (UniqueName: \"kubernetes.io/projected/7a720020-91b0-4d87-97f5-c8418dd80cd3-kube-api-access-n7kq7\") pod \"maas-api-revoke-sub-82ef682c-p767k\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:26.925636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:26.925553    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:27.251293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:27.251259    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"]
Oct 30 22:33:27.252060 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:33:27.252026    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a720020_91b0_4d87_97f5_c8418dd80cd3.slice/crio-bf04470ffc2ec51c26111ee7d6ec3be547039d1a5c65977227c914670199db45 WatchSource:0}: Error finding container bf04470ffc2ec51c26111ee7d6ec3be547039d1a5c65977227c914670199db45: Status 404 returned error can't find the container with id bf04470ffc2ec51c26111ee7d6ec3be547039d1a5c65977227c914670199db45
Oct 30 22:33:28.068022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:28.067934    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k" event={"ID":"7a720020-91b0-4d87-97f5-c8418dd80cd3","Type":"ContainerStarted","Data":"0394544b036e595088060280d3f6fd15d7b55812e98a12d3244f4e096a6d6329"}
Oct 30 22:33:28.068022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:28.067989    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k" event={"ID":"7a720020-91b0-4d87-97f5-c8418dd80cd3","Type":"ContainerStarted","Data":"bf04470ffc2ec51c26111ee7d6ec3be547039d1a5c65977227c914670199db45"}
Oct 30 22:33:28.101923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:28.101868    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k" podStartSLOduration=2.101852024 podStartE2EDuration="2.101852024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:28.101025312 +0000 UTC m=+1592.139964199" watchObservedRunningTime="2026-10-30 22:33:28.101852024 +0000 UTC m=+1592.140790911"
Oct 30 22:33:29.081026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:29.080989    4581 generic.go:356] "Generic (PLEG): container finished" podID="7a720020-91b0-4d87-97f5-c8418dd80cd3" containerID="0394544b036e595088060280d3f6fd15d7b55812e98a12d3244f4e096a6d6329" exitCode=0
Oct 30 22:33:29.081483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:29.081081    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k" event={"ID":"7a720020-91b0-4d87-97f5-c8418dd80cd3","Type":"ContainerDied","Data":"0394544b036e595088060280d3f6fd15d7b55812e98a12d3244f4e096a6d6329"}
Oct 30 22:33:30.311809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.311781    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:30.429529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.429495    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-openshift-ca\") pod \"7a720020-91b0-4d87-97f5-c8418dd80cd3\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") "
Oct 30 22:33:30.429735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.429583    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-xks-ca\") pod \"7a720020-91b0-4d87-97f5-c8418dd80cd3\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") "
Oct 30 22:33:30.429735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.429624    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a720020-91b0-4d87-97f5-c8418dd80cd3-kube-api-access-n7kq7\" (UniqueName: \"kubernetes.io/projected/7a720020-91b0-4d87-97f5-c8418dd80cd3-kube-api-access-n7kq7\") pod \"7a720020-91b0-4d87-97f5-c8418dd80cd3\" (UID: \"7a720020-91b0-4d87-97f5-c8418dd80cd3\") "
Oct 30 22:33:30.429960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.429929    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-openshift-ca" pod "7a720020-91b0-4d87-97f5-c8418dd80cd3" (UID: "7a720020-91b0-4d87-97f5-c8418dd80cd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:30.430319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.430300    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:30.431928 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.431903    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-xks-ca" pod "7a720020-91b0-4d87-97f5-c8418dd80cd3" (UID: "7a720020-91b0-4d87-97f5-c8418dd80cd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:30.431994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.431925    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a720020-91b0-4d87-97f5-c8418dd80cd3-kube-api-access-n7kq7" pod "7a720020-91b0-4d87-97f5-c8418dd80cd3" (UID: "7a720020-91b0-4d87-97f5-c8418dd80cd3"). InnerVolumeSpecName "kube-api-access-n7kq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:30.531803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.531766    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a720020-91b0-4d87-97f5-c8418dd80cd3-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:30.531803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:30.531797    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7kq7\" (UniqueName: \"kubernetes.io/projected/7a720020-91b0-4d87-97f5-c8418dd80cd3-kube-api-access-n7kq7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:31.106059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:31.106028    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"
Oct 30 22:33:31.106059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:31.106034    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k" event={"ID":"7a720020-91b0-4d87-97f5-c8418dd80cd3","Type":"ContainerDied","Data":"bf04470ffc2ec51c26111ee7d6ec3be547039d1a5c65977227c914670199db45"}
Oct 30 22:33:31.106410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:31.106093    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf04470ffc2ec51c26111ee7d6ec3be547039d1a5c65977227c914670199db45"
Oct 30 22:33:35.256886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:35.256845    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"]
Oct 30 22:33:35.277788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:35.277748    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-370ec2-jz629"]
Oct 30 22:33:36.099369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:36.099330    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b" path="/var/lib/kubelet/pods/608392f7-a5a1-47a8-ba9d-6b84f8b0bc7b/volumes"
Oct 30 22:33:40.821967 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:33:40.821927    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:33:41.112198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.112163    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"]
Oct 30 22:33:41.113020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.113005    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7a720020-91b0-4d87-97f5-c8418dd80cd3" containerName="revoke-keys"
Oct 30 22:33:41.113073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.113021    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a720020-91b0-4d87-97f5-c8418dd80cd3" containerName="revoke-keys"
Oct 30 22:33:41.113168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.113159    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a720020-91b0-4d87-97f5-c8418dd80cd3" containerName="revoke-keys"
Oct 30 22:33:41.117982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.117955    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.136840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.136812    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"]
Oct 30 22:33:41.139563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.139538    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:41.139739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.139719    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:33:41.139788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.139726    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:33:41.139788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.139725    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:33:41.210072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.210025    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.210324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.210276    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9px\" (UniqueName: \"kubernetes.io/projected/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-kube-api-access-gk9px\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.210590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.210556    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.312668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.312626    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9px\" (UniqueName: \"kubernetes.io/projected/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-kube-api-access-gk9px\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.312911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.312784    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.313066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.313046    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.313600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.313579    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.315719 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.315696    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.331195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.331172    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9px\" (UniqueName: \"kubernetes.io/projected/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-kube-api-access-gk9px\") pod \"maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.436825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.436745    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:41.772399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:41.772370    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"]
Oct 30 22:33:41.773938 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:33:41.773914    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a811bf0_3bdb_4b3b_9bd5_0100dfef137a.slice/crio-fccfe9b2d052fba2c5845754dddca6866e2cea5529e1ff2260814a4a0e76afc0 WatchSource:0}: Error finding container fccfe9b2d052fba2c5845754dddca6866e2cea5529e1ff2260814a4a0e76afc0: Status 404 returned error can't find the container with id fccfe9b2d052fba2c5845754dddca6866e2cea5529e1ff2260814a4a0e76afc0
Oct 30 22:33:42.333866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:42.333828    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb" event={"ID":"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a","Type":"ContainerStarted","Data":"131fc586682c96740ee9a2b8c626f2734839869cc1482c2068d78159bfe3b1ee"}
Oct 30 22:33:42.334382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:42.333915    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb" event={"ID":"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a","Type":"ContainerStarted","Data":"fccfe9b2d052fba2c5845754dddca6866e2cea5529e1ff2260814a4a0e76afc0"}
Oct 30 22:33:42.368763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:42.368711    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb" podStartSLOduration=1.368697096 podStartE2EDuration="1.368697096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:42.367087725 +0000 UTC m=+1606.406026614" watchObservedRunningTime="2026-10-30 22:33:42.368697096 +0000 UTC m=+1606.407636029"
Oct 30 22:33:42.991527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:42.991492    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"]
Oct 30 22:33:42.991895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:42.991816    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy" containerID="cri-o://208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1" gracePeriod=30
Oct 30 22:33:43.002716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.002677    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"]
Oct 30 22:33:43.002975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.002943    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy" containerID="cri-o://b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c" gracePeriod=30
Oct 30 22:33:43.011619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.011584    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"]
Oct 30 22:33:43.011866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.011835    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy" containerID="cri-o://44a5cd5113bf760b10507a789ecfd5cc36fdad2b442cddb0300aff685c4fc5a0" gracePeriod=30
Oct 30 22:33:43.408280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.407042    4581 generic.go:356] "Generic (PLEG): container finished" podID="0a811bf0-3bdb-4b3b-9bd5-0100dfef137a" containerID="131fc586682c96740ee9a2b8c626f2734839869cc1482c2068d78159bfe3b1ee" exitCode=0
Oct 30 22:33:43.408280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.407320    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb" event={"ID":"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a","Type":"ContainerDied","Data":"131fc586682c96740ee9a2b8c626f2734839869cc1482c2068d78159bfe3b1ee"}
Oct 30 22:33:43.604131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.604091    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"]
Oct 30 22:33:43.610411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.610386    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.627933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.627402    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"]
Oct 30 22:33:43.708386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.708298    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.708540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.708478    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.708598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.708583    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlhf\" (UniqueName: \"kubernetes.io/projected/a5389fc3-5275-4114-8a2b-61d99b0f53bc-kube-api-access-ljlhf\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.810281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.810245    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.810462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.810361    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlhf\" (UniqueName: \"kubernetes.io/projected/a5389fc3-5275-4114-8a2b-61d99b0f53bc-kube-api-access-ljlhf\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.810566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.810547    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.811231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.811162    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.812978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.812954    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.827742 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.827720    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlhf\" (UniqueName: \"kubernetes.io/projected/a5389fc3-5275-4114-8a2b-61d99b0f53bc-kube-api-access-ljlhf\") pod \"maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:43.928367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:43.928333    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:44.241696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.241632    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"]
Oct 30 22:33:44.242356 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:33:44.242328    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5389fc3_5275_4114_8a2b_61d99b0f53bc.slice/crio-23777314de75c8a9157389249dde5d5d0ba229f76a07d10b71dc0c7da264a627 WatchSource:0}: Error finding container 23777314de75c8a9157389249dde5d5d0ba229f76a07d10b71dc0c7da264a627: Status 404 returned error can't find the container with id 23777314de75c8a9157389249dde5d5d0ba229f76a07d10b71dc0c7da264a627
Oct 30 22:33:44.418502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.418470    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g" event={"ID":"a5389fc3-5275-4114-8a2b-61d99b0f53bc","Type":"ContainerStarted","Data":"23777314de75c8a9157389249dde5d5d0ba229f76a07d10b71dc0c7da264a627"}
Oct 30 22:33:44.608494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.608467    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:44.626435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.626405    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-kube-api-access-gk9px\" (UniqueName: \"kubernetes.io/projected/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-kube-api-access-gk9px\") pod \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") "
Oct 30 22:33:44.626635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.626487    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-openshift-ca\") pod \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") "
Oct 30 22:33:44.626707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.626637    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-xks-ca\") pod \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\" (UID: \"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a\") "
Oct 30 22:33:44.627852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.627820    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-openshift-ca" pod "0a811bf0-3bdb-4b3b-9bd5-0100dfef137a" (UID: "0a811bf0-3bdb-4b3b-9bd5-0100dfef137a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:44.632491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.632466    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-xks-ca" pod "0a811bf0-3bdb-4b3b-9bd5-0100dfef137a" (UID: "0a811bf0-3bdb-4b3b-9bd5-0100dfef137a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:44.633996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.633971    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-kube-api-access-gk9px" pod "0a811bf0-3bdb-4b3b-9bd5-0100dfef137a" (UID: "0a811bf0-3bdb-4b3b-9bd5-0100dfef137a"). InnerVolumeSpecName "kube-api-access-gk9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:44.729875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.729843    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:44.729875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.729873    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk9px\" (UniqueName: \"kubernetes.io/projected/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-kube-api-access-gk9px\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:44.729875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.729883    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:44.828487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.828448    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"]
Oct 30 22:33:44.829762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.829737    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0a811bf0-3bdb-4b3b-9bd5-0100dfef137a" containerName="revoke-keys"
Oct 30 22:33:44.829896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.829764    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a811bf0-3bdb-4b3b-9bd5-0100dfef137a" containerName="revoke-keys"
Oct 30 22:33:44.830147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.830123    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a811bf0-3bdb-4b3b-9bd5-0100dfef137a" containerName="revoke-keys"
Oct 30 22:33:44.834875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.834850    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:44.851168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.851138    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bdd94b\""
Oct 30 22:33:44.855487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.855462    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"]
Oct 30 22:33:44.867626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.867593    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bdd94b-dockercfg-f2gv4\""
Oct 30 22:33:44.947296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.939118    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86dw\" (UniqueName: \"kubernetes.io/projected/841bf7d5-6a07-4d7c-a170-bd39df91d349-kube-api-access-k86dw\") pod \"payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl\" (UID: \"841bf7d5-6a07-4d7c-a170-bd39df91d349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:44.947296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:44.941129    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841bf7d5-6a07-4d7c-a170-bd39df91d349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl\" (UID: \"841bf7d5-6a07-4d7c-a170-bd39df91d349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:45.099409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.092780    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86dw\" (UniqueName: \"kubernetes.io/projected/841bf7d5-6a07-4d7c-a170-bd39df91d349-kube-api-access-k86dw\") pod \"payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl\" (UID: \"841bf7d5-6a07-4d7c-a170-bd39df91d349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:45.099409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.092904    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841bf7d5-6a07-4d7c-a170-bd39df91d349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl\" (UID: \"841bf7d5-6a07-4d7c-a170-bd39df91d349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:45.099409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.093951    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841bf7d5-6a07-4d7c-a170-bd39df91d349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl\" (UID: \"841bf7d5-6a07-4d7c-a170-bd39df91d349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:45.162487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.162446    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"]
Oct 30 22:33:45.277868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.277839    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:45.309986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.309629    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"]
Oct 30 22:33:45.321638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.320957    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86dw\" (UniqueName: \"kubernetes.io/projected/841bf7d5-6a07-4d7c-a170-bd39df91d349-kube-api-access-k86dw\") pod \"payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl\" (UID: \"841bf7d5-6a07-4d7c-a170-bd39df91d349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:45.405687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.405660    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42s7\" (UniqueName: \"kubernetes.io/projected/520d7bde-53bf-42a2-a31a-17555e1981a8-kube-api-access-m42s7\") pod \"payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q\" (UID: \"520d7bde-53bf-42a2-a31a-17555e1981a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:45.406033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.406015    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520d7bde-53bf-42a2-a31a-17555e1981a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q\" (UID: \"520d7bde-53bf-42a2-a31a-17555e1981a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:45.429156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.429126    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"
Oct 30 22:33:45.429619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.429151    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb" event={"ID":"0a811bf0-3bdb-4b3b-9bd5-0100dfef137a","Type":"ContainerDied","Data":"fccfe9b2d052fba2c5845754dddca6866e2cea5529e1ff2260814a4a0e76afc0"}
Oct 30 22:33:45.429619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.429203    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fccfe9b2d052fba2c5845754dddca6866e2cea5529e1ff2260814a4a0e76afc0"
Oct 30 22:33:45.431084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.431059    4581 generic.go:356] "Generic (PLEG): container finished" podID="a5389fc3-5275-4114-8a2b-61d99b0f53bc" containerID="075a7ca4706bef336f4147e51ec89b0bda3d9e5185c90541dc1eeb8dbfcecbee" exitCode=0
Oct 30 22:33:45.431236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.431118    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g" event={"ID":"a5389fc3-5275-4114-8a2b-61d99b0f53bc","Type":"ContainerDied","Data":"075a7ca4706bef336f4147e51ec89b0bda3d9e5185c90541dc1eeb8dbfcecbee"}
Oct 30 22:33:45.463063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.463040    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:45.507579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.507545    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42s7\" (UniqueName: \"kubernetes.io/projected/520d7bde-53bf-42a2-a31a-17555e1981a8-kube-api-access-m42s7\") pod \"payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q\" (UID: \"520d7bde-53bf-42a2-a31a-17555e1981a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:45.507794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.507758    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520d7bde-53bf-42a2-a31a-17555e1981a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q\" (UID: \"520d7bde-53bf-42a2-a31a-17555e1981a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:45.508489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.508468    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520d7bde-53bf-42a2-a31a-17555e1981a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q\" (UID: \"520d7bde-53bf-42a2-a31a-17555e1981a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:45.524521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.524493    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42s7\" (UniqueName: \"kubernetes.io/projected/520d7bde-53bf-42a2-a31a-17555e1981a8-kube-api-access-m42s7\") pod \"payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q\" (UID: \"520d7bde-53bf-42a2-a31a-17555e1981a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:45.652749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.652664    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:45.801644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.801323    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"]
Oct 30 22:33:45.803195 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:33:45.803168    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841bf7d5_6a07_4d7c_a170_bd39df91d349.slice/crio-5620e9bb47afd8f19132d61740f1653f9b6f6031c1bf30b02f9f447e5356ef17 WatchSource:0}: Error finding container 5620e9bb47afd8f19132d61740f1653f9b6f6031c1bf30b02f9f447e5356ef17: Status 404 returned error can't find the container with id 5620e9bb47afd8f19132d61740f1653f9b6f6031c1bf30b02f9f447e5356ef17
Oct 30 22:33:45.970166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:45.970140    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"]
Oct 30 22:33:45.973238 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:33:45.973180    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520d7bde_53bf_42a2_a31a_17555e1981a8.slice/crio-ec95b09eae8ef9df5ed2ec859261390dd2d06268ee69511742e47bab22fd3766 WatchSource:0}: Error finding container ec95b09eae8ef9df5ed2ec859261390dd2d06268ee69511742e47bab22fd3766: Status 404 returned error can't find the container with id ec95b09eae8ef9df5ed2ec859261390dd2d06268ee69511742e47bab22fd3766
Oct 30 22:33:46.443827 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.443776    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q" event={"ID":"520d7bde-53bf-42a2-a31a-17555e1981a8","Type":"ContainerStarted","Data":"de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a"}
Oct 30 22:33:46.444337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.443850    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q" event={"ID":"520d7bde-53bf-42a2-a31a-17555e1981a8","Type":"ContainerStarted","Data":"ec95b09eae8ef9df5ed2ec859261390dd2d06268ee69511742e47bab22fd3766"}
Oct 30 22:33:46.444337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.443880    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:46.446160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.446128    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl" event={"ID":"841bf7d5-6a07-4d7c-a170-bd39df91d349","Type":"ContainerStarted","Data":"1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d"}
Oct 30 22:33:46.446332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.446183    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl" event={"ID":"841bf7d5-6a07-4d7c-a170-bd39df91d349","Type":"ContainerStarted","Data":"5620e9bb47afd8f19132d61740f1653f9b6f6031c1bf30b02f9f447e5356ef17"}
Oct 30 22:33:46.446332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.446265    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:46.481111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.481052    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q" podStartSLOduration=2.481034486 podStartE2EDuration="2.481034486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:46.476310057 +0000 UTC m=+1610.515248944" watchObservedRunningTime="2026-10-30 22:33:46.481034486 +0000 UTC m=+1610.519973371"
Oct 30 22:33:46.514580 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.514531    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl" podStartSLOduration=2.514516737 podStartE2EDuration="2.514516737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:46.508651648 +0000 UTC m=+1610.547590535" watchObservedRunningTime="2026-10-30 22:33:46.514516737 +0000 UTC m=+1610.553455652"
Oct 30 22:33:46.745341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.745306    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:46.848462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.848412    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5389fc3-5275-4114-8a2b-61d99b0f53bc-kube-api-access-ljlhf\" (UniqueName: \"kubernetes.io/projected/a5389fc3-5275-4114-8a2b-61d99b0f53bc-kube-api-access-ljlhf\") pod \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") "
Oct 30 22:33:46.848672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.848481    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-xks-ca\") pod \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") "
Oct 30 22:33:46.848672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.848624    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-openshift-ca\") pod \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\" (UID: \"a5389fc3-5275-4114-8a2b-61d99b0f53bc\") "
Oct 30 22:33:46.849109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.849078    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-openshift-ca" pod "a5389fc3-5275-4114-8a2b-61d99b0f53bc" (UID: "a5389fc3-5275-4114-8a2b-61d99b0f53bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:46.849476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.849451    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:46.851535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.851505    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-xks-ca" pod "a5389fc3-5275-4114-8a2b-61d99b0f53bc" (UID: "a5389fc3-5275-4114-8a2b-61d99b0f53bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:46.851643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.851555    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5389fc3-5275-4114-8a2b-61d99b0f53bc-kube-api-access-ljlhf" pod "a5389fc3-5275-4114-8a2b-61d99b0f53bc" (UID: "a5389fc3-5275-4114-8a2b-61d99b0f53bc"). InnerVolumeSpecName "kube-api-access-ljlhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:46.951457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.951419    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlhf\" (UniqueName: \"kubernetes.io/projected/a5389fc3-5275-4114-8a2b-61d99b0f53bc-kube-api-access-ljlhf\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:46.951457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:46.951451    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5389fc3-5275-4114-8a2b-61d99b0f53bc-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:47.459817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:47.459782    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"
Oct 30 22:33:47.459817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:47.459815    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g" event={"ID":"a5389fc3-5275-4114-8a2b-61d99b0f53bc","Type":"ContainerDied","Data":"23777314de75c8a9157389249dde5d5d0ba229f76a07d10b71dc0c7da264a627"}
Oct 30 22:33:47.460396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:47.459865    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23777314de75c8a9157389249dde5d5d0ba229f76a07d10b71dc0c7da264a627"
Oct 30 22:33:48.465926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.465893    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:33:48.476729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.476698    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:33:48.482511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.480129    4581 generic.go:356] "Generic (PLEG): container finished" podID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerID="b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c" exitCode=0
Oct 30 22:33:48.482511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.480345    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" event={"ID":"32e5b32e-12f7-4449-887b-e73e9b332b63","Type":"ContainerDied","Data":"b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c"}
Oct 30 22:33:48.482511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.480394    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x" event={"ID":"32e5b32e-12f7-4449-887b-e73e9b332b63","Type":"ContainerDied","Data":"778d90107c7dc111d87c17926fcd7560e571df5c608d9c9ce1b3d6ee5ed1fd44"}
Oct 30 22:33:48.482511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.480423    4581 scope.go:122] "RemoveContainer" containerID="b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c"
Oct 30 22:33:48.482511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.480588    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"
Oct 30 22:33:48.504510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.504474    4581 generic.go:356] "Generic (PLEG): container finished" podID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerID="208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1" exitCode=0
Oct 30 22:33:48.504710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.504589    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"
Oct 30 22:33:48.504710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.504662    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" event={"ID":"f34b75f1-c64b-444d-94ad-c5f7798bf66f","Type":"ContainerDied","Data":"208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1"}
Oct 30 22:33:48.504848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.504712    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6" event={"ID":"f34b75f1-c64b-444d-94ad-c5f7798bf66f","Type":"ContainerDied","Data":"1894b3b2f94437a2aab9384ab7a485a9fbfd27cac2cd336430892f310bd72291"}
Oct 30 22:33:48.509882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.509788    4581 generic.go:356] "Generic (PLEG): container finished" podID="79545781-49d9-4fba-be6c-25548aabcccf" containerID="44a5cd5113bf760b10507a789ecfd5cc36fdad2b442cddb0300aff685c4fc5a0" exitCode=0
Oct 30 22:33:48.509882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.509824    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" event={"ID":"79545781-49d9-4fba-be6c-25548aabcccf","Type":"ContainerDied","Data":"44a5cd5113bf760b10507a789ecfd5cc36fdad2b442cddb0300aff685c4fc5a0"}
Oct 30 22:33:48.516710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.516690    4581 scope.go:122] "RemoveContainer" containerID="b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c"
Oct 30 22:33:48.518563 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:33:48.518531    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c\": container with ID starting with b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c not found: ID does not exist" containerID="b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c"
Oct 30 22:33:48.518691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.518575    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c"} err="failed to get container status \"b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c\": rpc error: code = NotFound desc = could not find container \"b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c\": container with ID starting with b3ddecd076bec515c86644e50620b436b5a7c702ddfd70091662a9b85273ee3c not found: ID does not exist"
Oct 30 22:33:48.518691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.518599    4581 scope.go:122] "RemoveContainer" containerID="208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1"
Oct 30 22:33:48.538675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.538651    4581 scope.go:122] "RemoveContainer" containerID="208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1"
Oct 30 22:33:48.539622 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:33:48.539198    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1\": container with ID starting with 208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1 not found: ID does not exist" containerID="208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1"
Oct 30 22:33:48.539781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.539755    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1"} err="failed to get container status \"208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1\": rpc error: code = NotFound desc = could not find container \"208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1\": container with ID starting with 208a4b9d8f2cf28c3c4dcc0126744983558cbe07849e2663abaec0e3ded25cb1 not found: ID does not exist"
Oct 30 22:33:48.557295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.554640    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:33:48.602846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.602812    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-kube-api-access-qs6qt\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603361    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-envoy\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603426    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-socket\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603464    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istiod-ca-cert\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603533    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-credential-socket\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603607    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-data\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603633    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-podinfo\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603659    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-socket\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603702    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-kube-api-access-cssdr\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-kube-api-access-cssdr\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603907    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-istio-token\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-istio-token\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603941    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-certs\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603984    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-socket\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.603986    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-credential-socket" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604025    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-envoy\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604061    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-credential-socket\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604173    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-token\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-token\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604260    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-certs\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604301    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79545781-49d9-4fba-be6c-25548aabcccf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79545781-49d9-4fba-be6c-25548aabcccf-istiod-ca-cert\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604330    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e5b32e-12f7-4449-887b-e73e9b332b63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e5b32e-12f7-4449-887b-e73e9b332b63-istiod-ca-cert\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604387    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-envoy\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604411    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-token\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-token\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604472    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-kube-api-access-nhr5l\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-kube-api-access-nhr5l\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604522    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79545781-49d9-4fba-be6c-25548aabcccf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79545781-49d9-4fba-be6c-25548aabcccf-istio-podinfo\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604551    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-data\") pod \"32e5b32e-12f7-4449-887b-e73e9b332b63\" (UID: \"32e5b32e-12f7-4449-887b-e73e9b332b63\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604605    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-credential-socket\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.604632    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-podinfo\") pod \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\" (UID: \"f34b75f1-c64b-444d-94ad-c5f7798bf66f\") "
Oct 30 22:33:48.607564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.606517    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.609508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.607656    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-socket" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.609508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.608010    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istiod-ca-cert" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.609508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.608321    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-socket" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.609508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.609468    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-data" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.609801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.609772    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-data" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.610123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.609981    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-socket" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.610229 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.610179    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-certs" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.611335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.611309    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79545781-49d9-4fba-be6c-25548aabcccf-istiod-ca-cert" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.611657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.611636    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-certs" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.613423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.612758    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-credential-socket" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.613423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.613117    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e5b32e-12f7-4449-887b-e73e9b332b63-istiod-ca-cert" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.613423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.613130    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-credential-socket" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.616897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.616837    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-podinfo" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.621703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.617535    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-kube-api-access-nhr5l" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "kube-api-access-nhr5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.621703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.621477    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-envoy" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.621703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.621582    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-envoy" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.621703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.621671    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79545781-49d9-4fba-be6c-25548aabcccf-istio-podinfo" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.621972 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.621706    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-podinfo" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.621972 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.621717    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-kube-api-access-cssdr" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "kube-api-access-cssdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.621972 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.621744    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-token" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.626410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.625613    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-envoy" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.626410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.625774    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-istio-token" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.626410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.626361    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-kube-api-access-qs6qt" pod "32e5b32e-12f7-4449-887b-e73e9b332b63" (UID: "32e5b32e-12f7-4449-887b-e73e9b332b63"). InnerVolumeSpecName "kube-api-access-qs6qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.626973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.626943    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-token" pod "f34b75f1-c64b-444d-94ad-c5f7798bf66f" (UID: "f34b75f1-c64b-444d-94ad-c5f7798bf66f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.708302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.707681    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-data\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.708302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.707765    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-certs\") pod \"79545781-49d9-4fba-be6c-25548aabcccf\" (UID: \"79545781-49d9-4fba-be6c-25548aabcccf\") "
Oct 30 22:33:48.708583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.708484    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-data" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.708583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.708572    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-certs" pod "79545781-49d9-4fba-be6c-25548aabcccf" (UID: "79545781-49d9-4fba-be6c-25548aabcccf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.709077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709048    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709084    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709100    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709113    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709125    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709137    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709150    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709162    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709176    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cssdr\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-kube-api-access-cssdr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709189    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709202    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709725    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709756    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709772    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709785    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709798    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709811    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79545781-49d9-4fba-be6c-25548aabcccf-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709823    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32e5b32e-12f7-4449-887b-e73e9b332b63-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709837    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79545781-49d9-4fba-be6c-25548aabcccf-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709850    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709863    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhr5l\" (UniqueName: \"kubernetes.io/projected/79545781-49d9-4fba-be6c-25548aabcccf-kube-api-access-nhr5l\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709876    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79545781-49d9-4fba-be6c-25548aabcccf-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709890    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32e5b32e-12f7-4449-887b-e73e9b332b63-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709903    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34b75f1-c64b-444d-94ad-c5f7798bf66f-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709916    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34b75f1-c64b-444d-94ad-c5f7798bf66f-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.709932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.709932    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/32e5b32e-12f7-4449-887b-e73e9b332b63-kube-api-access-qs6qt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.833752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.833718    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"]
Oct 30 22:33:48.838963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.838938    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-j564x"]
Oct 30 22:33:48.866033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.866001    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"]
Oct 30 22:33:48.873183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:48.873155    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2mvj6"]
Oct 30 22:33:49.525169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:49.525130    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"
Oct 30 22:33:49.525617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:49.525141    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss" event={"ID":"79545781-49d9-4fba-be6c-25548aabcccf","Type":"ContainerDied","Data":"201d3f2a9615f06a33151b345b78f8d8301e1d652f97b56e1048a36af3d1eccd"}
Oct 30 22:33:49.525617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:49.525318    4581 scope.go:122] "RemoveContainer" containerID="44a5cd5113bf760b10507a789ecfd5cc36fdad2b442cddb0300aff685c4fc5a0"
Oct 30 22:33:49.574526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:49.574494    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"]
Oct 30 22:33:49.584090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:49.582479    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-xwkss"]
Oct 30 22:33:50.108682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:50.108644    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" path="/var/lib/kubelet/pods/32e5b32e-12f7-4449-887b-e73e9b332b63/volumes"
Oct 30 22:33:50.109410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:50.109390    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79545781-49d9-4fba-be6c-25548aabcccf" path="/var/lib/kubelet/pods/79545781-49d9-4fba-be6c-25548aabcccf/volumes"
Oct 30 22:33:50.110027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:50.110011    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" path="/var/lib/kubelet/pods/f34b75f1-c64b-444d-94ad-c5f7798bf66f/volumes"
Oct 30 22:33:51.580598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:51.580562    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"]
Oct 30 22:33:51.614179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:51.614151    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"]
Oct 30 22:33:52.315074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:52.315040    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"]
Oct 30 22:33:52.323054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:52.323024    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f443d6-8zcgw"]
Oct 30 22:33:54.103138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:54.103099    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bad6af9-d379-4152-9fd1-2679605e6cc3" path="/var/lib/kubelet/pods/1bad6af9-d379-4152-9fd1-2679605e6cc3/volumes"
Oct 30 22:33:56.581970 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:56.581884    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs" podUID="e7774cd9-8f15-45ad-9372-521c16e5a330" containerName="payload-processing" containerID="cri-o://b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8" gracePeriod=25
Oct 30 22:33:56.615477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:56.614684    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8" podUID="9525c8e4-f28e-4563-9147-9d8cb7e01e02" containerName="payload-pre-processing" containerID="cri-o://f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f" gracePeriod=25
Oct 30 22:33:56.950849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:56.950817    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:33:56.985789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:56.985761    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:33:57.065023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.064983    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7774cd9-8f15-45ad-9372-521c16e5a330-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7774cd9-8f15-45ad-9372-521c16e5a330-plugins-config-volume\") pod \"e7774cd9-8f15-45ad-9372-521c16e5a330\" (UID: \"e7774cd9-8f15-45ad-9372-521c16e5a330\") "
Oct 30 22:33:57.065299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.065044    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7774cd9-8f15-45ad-9372-521c16e5a330-kube-api-access-s2gcd\" (UniqueName: \"kubernetes.io/projected/e7774cd9-8f15-45ad-9372-521c16e5a330-kube-api-access-s2gcd\") pod \"e7774cd9-8f15-45ad-9372-521c16e5a330\" (UID: \"e7774cd9-8f15-45ad-9372-521c16e5a330\") "
Oct 30 22:33:57.065871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.065823    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7774cd9-8f15-45ad-9372-521c16e5a330-plugins-config-volume" pod "e7774cd9-8f15-45ad-9372-521c16e5a330" (UID: "e7774cd9-8f15-45ad-9372-521c16e5a330"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:57.068796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.068767    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7774cd9-8f15-45ad-9372-521c16e5a330-kube-api-access-s2gcd" pod "e7774cd9-8f15-45ad-9372-521c16e5a330" (UID: "e7774cd9-8f15-45ad-9372-521c16e5a330"). InnerVolumeSpecName "kube-api-access-s2gcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:57.167538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.167447    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9525c8e4-f28e-4563-9147-9d8cb7e01e02-kube-api-access-w9glb\" (UniqueName: \"kubernetes.io/projected/9525c8e4-f28e-4563-9147-9d8cb7e01e02-kube-api-access-w9glb\") pod \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\" (UID: \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\") "
Oct 30 22:33:57.167709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.167581    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9525c8e4-f28e-4563-9147-9d8cb7e01e02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525c8e4-f28e-4563-9147-9d8cb7e01e02-plugins-config-volume\") pod \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\" (UID: \"9525c8e4-f28e-4563-9147-9d8cb7e01e02\") "
Oct 30 22:33:57.168054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.168021    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9525c8e4-f28e-4563-9147-9d8cb7e01e02-plugins-config-volume" pod "9525c8e4-f28e-4563-9147-9d8cb7e01e02" (UID: "9525c8e4-f28e-4563-9147-9d8cb7e01e02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:57.168489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.168468    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525c8e4-f28e-4563-9147-9d8cb7e01e02-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:57.168567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.168491    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7774cd9-8f15-45ad-9372-521c16e5a330-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:57.168567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.168501    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gcd\" (UniqueName: \"kubernetes.io/projected/e7774cd9-8f15-45ad-9372-521c16e5a330-kube-api-access-s2gcd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:57.170227 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.170187    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9525c8e4-f28e-4563-9147-9d8cb7e01e02-kube-api-access-w9glb" pod "9525c8e4-f28e-4563-9147-9d8cb7e01e02" (UID: "9525c8e4-f28e-4563-9147-9d8cb7e01e02"). InnerVolumeSpecName "kube-api-access-w9glb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:57.270679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.270631    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9glb\" (UniqueName: \"kubernetes.io/projected/9525c8e4-f28e-4563-9147-9d8cb7e01e02-kube-api-access-w9glb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:33:57.464153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.464060    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:33:57.464369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.464159    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:33:57.669426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.669388    4581 generic.go:356] "Generic (PLEG): container finished" podID="e7774cd9-8f15-45ad-9372-521c16e5a330" containerID="b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8" exitCode=0
Oct 30 22:33:57.669959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.669455    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"
Oct 30 22:33:57.669959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.669476    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs" event={"ID":"e7774cd9-8f15-45ad-9372-521c16e5a330","Type":"ContainerDied","Data":"b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8"}
Oct 30 22:33:57.669959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.669567    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs" event={"ID":"e7774cd9-8f15-45ad-9372-521c16e5a330","Type":"ContainerDied","Data":"d7c13e186664749981fb33259fff1c17827fcc6de437ed04bc7d28215ba290ed"}
Oct 30 22:33:57.669959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.669614    4581 scope.go:122] "RemoveContainer" containerID="b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8"
Oct 30 22:33:57.672922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.672741    4581 generic.go:356] "Generic (PLEG): container finished" podID="9525c8e4-f28e-4563-9147-9d8cb7e01e02" containerID="f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f" exitCode=0
Oct 30 22:33:57.672922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.672908    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8" event={"ID":"9525c8e4-f28e-4563-9147-9d8cb7e01e02","Type":"ContainerDied","Data":"f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f"}
Oct 30 22:33:57.673693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.672963    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8" event={"ID":"9525c8e4-f28e-4563-9147-9d8cb7e01e02","Type":"ContainerDied","Data":"bcc124262e73bd64a632d89e1555fcf137017b6437f10c6c4534c2dda6745e44"}
Oct 30 22:33:57.673693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.673032    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"
Oct 30 22:33:57.687139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.687112    4581 scope.go:122] "RemoveContainer" containerID="b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8"
Oct 30 22:33:57.687572 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:33:57.687549    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8\": container with ID starting with b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8 not found: ID does not exist" containerID="b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8"
Oct 30 22:33:57.687647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.687586    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8"} err="failed to get container status \"b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8\": rpc error: code = NotFound desc = could not find container \"b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8\": container with ID starting with b213693f4019f05e630b0f7c340d3b48bd8ab534d3da72d544f17d484184eab8 not found: ID does not exist"
Oct 30 22:33:57.687647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.687613    4581 scope.go:122] "RemoveContainer" containerID="f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f"
Oct 30 22:33:57.699279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.699260    4581 scope.go:122] "RemoveContainer" containerID="f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f"
Oct 30 22:33:57.699605 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:33:57.699584    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f\": container with ID starting with f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f not found: ID does not exist" containerID="f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f"
Oct 30 22:33:57.699703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.699612    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f"} err="failed to get container status \"f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f\": rpc error: code = NotFound desc = could not find container \"f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f\": container with ID starting with f8aae44e0475e524c6b0de4a6c83782bed5bbdb308dd6d0a921be36798fe0e0f not found: ID does not exist"
Oct 30 22:33:57.728484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.728392    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"]
Oct 30 22:33:57.734266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.734205    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30edd7-57c7586d97-zmlbs"]
Oct 30 22:33:57.765878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.765434    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"]
Oct 30 22:33:57.783471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:57.783435    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30edd7-5699985987-rczc8"]
Oct 30 22:33:58.029882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:58.029793    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"]
Oct 30 22:33:58.034595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:58.034560    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cd855d-bmx2d"]
Oct 30 22:33:58.100783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:58.100747    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55332ab2-75b3-4d97-a454-6f63e404af4d" path="/var/lib/kubelet/pods/55332ab2-75b3-4d97-a454-6f63e404af4d/volumes"
Oct 30 22:33:58.101377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:58.101354    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9525c8e4-f28e-4563-9147-9d8cb7e01e02" path="/var/lib/kubelet/pods/9525c8e4-f28e-4563-9147-9d8cb7e01e02/volumes"
Oct 30 22:33:58.101875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:58.101855    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7774cd9-8f15-45ad-9372-521c16e5a330" path="/var/lib/kubelet/pods/e7774cd9-8f15-45ad-9372-521c16e5a330/volumes"
Oct 30 22:33:59.230073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:59.230038    4581 scope.go:122] "RemoveContainer" containerID="e7cc11095eb109f88ac6ce9207bbcd1f9ac14555b09358e132bb825c2cdbd8ed"
Oct 30 22:33:59.246998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:33:59.246974    4581 scope.go:122] "RemoveContainer" containerID="8c04472680aec1307149f6ba963ad16021266c4e54e5d37e3d192e2575a07110"
Oct 30 22:34:00.058697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.058635    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"]
Oct 30 22:34:00.059463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.059429    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" containerID="cri-o://099bcf2d5fb570a442bff8b1a4cf223ea91a23a85672570d7ad33c2f19ad9114" gracePeriod=30
Oct 30 22:34:00.645054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645019    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"]
Oct 30 22:34:00.645795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645779    4581 cpu_manager.go:414] "containerMap: removing container" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy"
Oct 30 22:34:00.645843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645797    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy"
Oct 30 22:34:00.645843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645823    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e7774cd9-8f15-45ad-9372-521c16e5a330" containerName="payload-processing"
Oct 30 22:34:00.645843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645827    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7774cd9-8f15-45ad-9372-521c16e5a330" containerName="payload-processing"
Oct 30 22:34:00.645954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645846    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9525c8e4-f28e-4563-9147-9d8cb7e01e02" containerName="payload-pre-processing"
Oct 30 22:34:00.645954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645852    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9525c8e4-f28e-4563-9147-9d8cb7e01e02" containerName="payload-pre-processing"
Oct 30 22:34:00.645954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645863    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy"
Oct 30 22:34:00.645954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645867    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy"
Oct 30 22:34:00.645954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645877    4581 cpu_manager.go:414] "containerMap: removing container" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy"
Oct 30 22:34:00.645954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645881    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy"
Oct 30 22:34:00.645954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645904    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a5389fc3-5275-4114-8a2b-61d99b0f53bc" containerName="revoke-keys"
Oct 30 22:34:00.645954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.645908    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5389fc3-5275-4114-8a2b-61d99b0f53bc" containerName="revoke-keys"
Oct 30 22:34:00.646269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.646009    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7774cd9-8f15-45ad-9372-521c16e5a330" containerName="payload-processing"
Oct 30 22:34:00.646269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.646022    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9525c8e4-f28e-4563-9147-9d8cb7e01e02" containerName="payload-pre-processing"
Oct 30 22:34:00.646269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.646030    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34b75f1-c64b-444d-94ad-c5f7798bf66f" containerName="istio-proxy"
Oct 30 22:34:00.646269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.646037    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="79545781-49d9-4fba-be6c-25548aabcccf" containerName="istio-proxy"
Oct 30 22:34:00.646269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.646047    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5389fc3-5275-4114-8a2b-61d99b0f53bc" containerName="revoke-keys"
Oct 30 22:34:00.646269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.646054    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e5b32e-12f7-4449-887b-e73e9b332b63" containerName="istio-proxy"
Oct 30 22:34:00.652030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.652002    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.664868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.664835    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:00.665022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.664902    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:00.673277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.673245    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"]
Oct 30 22:34:00.674083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.674057    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:00.674276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.674090    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:34:00.738907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.738874    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.739095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.738985    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbdvp\" (UniqueName: \"kubernetes.io/projected/bf619604-275f-4d75-af9a-009ea1ca0926-kube-api-access-hbdvp\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.739190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.739164    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.840467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.840420    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbdvp\" (UniqueName: \"kubernetes.io/projected/bf619604-275f-4d75-af9a-009ea1ca0926-kube-api-access-hbdvp\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.840651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.840536    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.840722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.840690    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.841419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.841339    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.843263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.843185    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.859768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.859735    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbdvp\" (UniqueName: \"kubernetes.io/projected/bf619604-275f-4d75-af9a-009ea1ca0926-kube-api-access-hbdvp\") pod \"maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:00.974230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:00.974125    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:01.308466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:01.307569    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"]
Oct 30 22:34:01.725659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:01.725620    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6" event={"ID":"bf619604-275f-4d75-af9a-009ea1ca0926","Type":"ContainerStarted","Data":"8830cda160fd7f81b154abfbc86f3b90b0cfc3841de30331b9c0f81220e18505"}
Oct 30 22:34:02.735869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:02.735784    4581 generic.go:356] "Generic (PLEG): container finished" podID="bf619604-275f-4d75-af9a-009ea1ca0926" containerID="1c4c6b2bbd43151c2b1518a3ed8dc879cc318f58333cd50ab418117d327f85b6" exitCode=0
Oct 30 22:34:02.736278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:02.735869    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6" event={"ID":"bf619604-275f-4d75-af9a-009ea1ca0926","Type":"ContainerDied","Data":"1c4c6b2bbd43151c2b1518a3ed8dc879cc318f58333cd50ab418117d327f85b6"}
Oct 30 22:34:03.716244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:03.704391    4581 patch_prober.go:29] interesting pod/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:34:03.716244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:03.704468    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:34:04.020933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.020897    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:04.035263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.035201    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"]
Oct 30 22:34:04.044519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.044489    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c65aa-786h2"]
Oct 30 22:34:04.110245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.109791    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5ba764-e1df-4205-aab3-2a79e9168fe6" path="/var/lib/kubelet/pods/4f5ba764-e1df-4205-aab3-2a79e9168fe6/volumes"
Oct 30 22:34:04.117363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.117304    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf619604-275f-4d75-af9a-009ea1ca0926-kube-api-access-hbdvp\" (UniqueName: \"kubernetes.io/projected/bf619604-275f-4d75-af9a-009ea1ca0926-kube-api-access-hbdvp\") pod \"bf619604-275f-4d75-af9a-009ea1ca0926\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") "
Oct 30 22:34:04.117649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.117368    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-openshift-ca\") pod \"bf619604-275f-4d75-af9a-009ea1ca0926\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") "
Oct 30 22:34:04.117649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.117554    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-xks-ca\") pod \"bf619604-275f-4d75-af9a-009ea1ca0926\" (UID: \"bf619604-275f-4d75-af9a-009ea1ca0926\") "
Oct 30 22:34:04.120040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.120012    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-openshift-ca" pod "bf619604-275f-4d75-af9a-009ea1ca0926" (UID: "bf619604-275f-4d75-af9a-009ea1ca0926"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:04.122325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.122201    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-xks-ca" pod "bf619604-275f-4d75-af9a-009ea1ca0926" (UID: "bf619604-275f-4d75-af9a-009ea1ca0926"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:04.185183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.185143    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf619604-275f-4d75-af9a-009ea1ca0926-kube-api-access-hbdvp" pod "bf619604-275f-4d75-af9a-009ea1ca0926" (UID: "bf619604-275f-4d75-af9a-009ea1ca0926"). InnerVolumeSpecName "kube-api-access-hbdvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:04.225499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.225413    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbdvp\" (UniqueName: \"kubernetes.io/projected/bf619604-275f-4d75-af9a-009ea1ca0926-kube-api-access-hbdvp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:04.225499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.225453    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:04.225499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.225470    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf619604-275f-4d75-af9a-009ea1ca0926-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:04.763664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.763622    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"
Oct 30 22:34:04.763664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.763659    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6" event={"ID":"bf619604-275f-4d75-af9a-009ea1ca0926","Type":"ContainerDied","Data":"8830cda160fd7f81b154abfbc86f3b90b0cfc3841de30331b9c0f81220e18505"}
Oct 30 22:34:04.763894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:04.763719    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8830cda160fd7f81b154abfbc86f3b90b0cfc3841de30331b9c0f81220e18505"
Oct 30 22:34:06.224243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.223803    4581 generic.go:356] "Generic (PLEG): container finished" podID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerID="099bcf2d5fb570a442bff8b1a4cf223ea91a23a85672570d7ad33c2f19ad9114" exitCode=0
Oct 30 22:34:06.249179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.249135    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" event={"ID":"c3c8d384-92f1-40f3-a233-84519c2ee72f","Type":"ContainerDied","Data":"099bcf2d5fb570a442bff8b1a4cf223ea91a23a85672570d7ad33c2f19ad9114"}
Oct 30 22:34:06.249925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.249901    4581 scope.go:122] "RemoveContainer" containerID="6051edd0e2f2dac4628d09b571d39054fb76b2ad37c7e8e90abd0082cd5ed4da"
Oct 30 22:34:06.265734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.255241    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:34:06.357239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.356605    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-credential-socket\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.357239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.356685    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-kube-api-access-d7npc\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-kube-api-access-d7npc\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.357239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.356729    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-data\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.357239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.356840    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.357239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.357030    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-token\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-token\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.357239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.357075    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c8d384-92f1-40f3-a233-84519c2ee72f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c8d384-92f1-40f3-a233-84519c2ee72f-istiod-ca-cert\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.357239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.357150    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-certs\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.357239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.357242    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-socket\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.357813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.357411    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-envoy\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.388253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.372243    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-socket" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:06.388253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.372583    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-certs" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:06.449493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.443333    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-data" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:06.449493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.443420    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-credential-socket" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:06.519461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.518741    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c8d384-92f1-40f3-a233-84519c2ee72f-istiod-ca-cert" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:06.537301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.529153    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:34:06.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.565618    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-token" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:06.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.567677    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:06.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.567728    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c8d384-92f1-40f3-a233-84519c2ee72f-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:06.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.567752    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:06.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.567765    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:06.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.567779    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:06.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.567809    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:06.583245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.568929    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:06.615302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.612743    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:06.615302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.614815    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:34:06.708275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.708209    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:06.725564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.723379    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo\") pod \"c3c8d384-92f1-40f3-a233-84519c2ee72f\" (UID: \"c3c8d384-92f1-40f3-a233-84519c2ee72f\") "
Oct 30 22:34:06.725994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.725951    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-kube-api-access-d7npc" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "kube-api-access-d7npc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:06.727678 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:34:06.727653    4581 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c3c8d384-92f1-40f3-a233-84519c2ee72f/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 30 22:34:06.727866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.727843    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:06.782243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.739511    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:34:06.782243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.751030    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:06.903510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.902783    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-envoy" pod "c3c8d384-92f1-40f3-a233-84519c2ee72f" (UID: "c3c8d384-92f1-40f3-a233-84519c2ee72f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:06.975756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.975653    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:06.975756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.975694    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c8d384-92f1-40f3-a233-84519c2ee72f-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:06.975756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:06.975711    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7npc\" (UniqueName: \"kubernetes.io/projected/c3c8d384-92f1-40f3-a233-84519c2ee72f-kube-api-access-d7npc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:07.349325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.346844    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv" event={"ID":"c3c8d384-92f1-40f3-a233-84519c2ee72f","Type":"ContainerDied","Data":"cce8a111cf098e3bab9448b15af1e2ffc261fe792e1130c3a752d3979f811083"}
Oct 30 22:34:07.349325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.346920    4581 scope.go:122] "RemoveContainer" containerID="099bcf2d5fb570a442bff8b1a4cf223ea91a23a85672570d7ad33c2f19ad9114"
Oct 30 22:34:07.349325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.347076    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"
Oct 30 22:34:07.511269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.508367    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"]
Oct 30 22:34:07.511269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.509560    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bf619604-275f-4d75-af9a-009ea1ca0926" containerName="revoke-keys"
Oct 30 22:34:07.511269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.509585    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf619604-275f-4d75-af9a-009ea1ca0926" containerName="revoke-keys"
Oct 30 22:34:07.511269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.509604    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy"
Oct 30 22:34:07.511269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.509612    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy"
Oct 30 22:34:07.511269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.509859    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy"
Oct 30 22:34:07.511269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.509880    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf619604-275f-4d75-af9a-009ea1ca0926" containerName="revoke-keys"
Oct 30 22:34:07.511269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.509891    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy"
Oct 30 22:34:07.569268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.561557    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:34:07.574909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.569945    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:34:07.574909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.570765    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.591483    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.591553    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.591608    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.591957    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.623313    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.623707    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.624297    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.624356    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.626915    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.626934    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.626951    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.627000    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.629368    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lnh72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:07.630447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.629427    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:07.677506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.677460    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvhx\" (UniqueName: \"kubernetes.io/projected/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-kube-api-access-kpvhx\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.678119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.678089    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.679370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.679347    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.705335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.704734    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 30 22:34:07.781769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.781731    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.806741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.806704    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.923346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.791400    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 22:34:07.923346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.792258    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.923346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.789394    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:34:07.923346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.873402    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:07.923346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.909012    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:34:07.948286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.925471    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:34:07.948286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.907290    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvhx\" (UniqueName: \"kubernetes.io/projected/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-kube-api-access-kpvhx\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.948286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.935657    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:07.965703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:07.965663    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"]
Oct 30 22:34:08.168435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.168387    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:08.301713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.301621    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:08.487963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.470355    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvhx\" (UniqueName: \"kubernetes.io/projected/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-kube-api-access-kpvhx\") pod \"maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:08.598472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.582855    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:34:08.598472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.583152    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:34:08.598472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.583385    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:34:08.598472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.583915    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:34:08.612382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.611862    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:08.640247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.612846    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:34:08.641869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.641330    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:34:08.641869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.641355    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:34:08.691243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.681945    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:08.868920 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:34:08.795261    4581 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c8d384_92f1_40f3_a233_84519c2ee72f.slice/crio-cce8a111cf098e3bab9448b15af1e2ffc261fe792e1130c3a752d3979f811083\": RecentStats: unable to find data in memory cache]"
Oct 30 22:34:08.917303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.910331    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:08.922755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.922710    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:34:08.923150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.923129    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:34:08.986687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:08.944012    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:09.008635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.008342    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"]
Oct 30 22:34:09.026278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.026171    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f443d6-openshift-default-6dc8b74cd7-nkzfv"]
Oct 30 22:34:09.072255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.071935    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:09.072255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.072177    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 30 22:34:09.224845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.224805    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:09.225576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.225545    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:34:09.318285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.309684    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:09.318285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.311327    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:34:09.342834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.342803    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 22:34:09.668251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.667904    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:09.720142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.719967    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:09.736957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.726733    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 22:34:09.754246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.738093    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 22:34:09.754612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.754591    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 22:34:09.754830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.754817    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 22:34:09.755017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.755004    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 22:34:09.756408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.756391    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 22:34:09.756636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.756621    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 22:34:09.756791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.742690    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:09.781257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.731637    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:09.833860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.831394    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 22:34:09.833860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.831452    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 22:34:09.894127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:09.894091    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 22:34:10.004124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:10.002761    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"]
Oct 30 22:34:10.110639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:10.110593    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" path="/var/lib/kubelet/pods/c3c8d384-92f1-40f3-a233-84519c2ee72f/volumes"
Oct 30 22:34:10.803586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:10.803551    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w" event={"ID":"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c","Type":"ContainerStarted","Data":"20f14a5373b34053f0620d65ddcd08cbfb3615366d1dde2f1195a064f4363d4d"}
Oct 30 22:34:10.804058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:10.803603    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w" event={"ID":"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c","Type":"ContainerStarted","Data":"7bc4acd4ab834e65c747f4baf39bb9fc093fd97f08c75c2d0fab3b09bd9a20c6"}
Oct 30 22:34:10.822401 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:34:10.822351    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:34:10.840440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:10.840331    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w" podStartSLOduration=3.840313618 podStartE2EDuration="3.840313618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:10.836443043 +0000 UTC m=+1634.875381929" watchObservedRunningTime="2026-10-30 22:34:10.840313618 +0000 UTC m=+1634.879252510"
Oct 30 22:34:11.816396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:11.816298    4581 generic.go:356] "Generic (PLEG): container finished" podID="1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c" containerID="20f14a5373b34053f0620d65ddcd08cbfb3615366d1dde2f1195a064f4363d4d" exitCode=0
Oct 30 22:34:11.816835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:11.816397    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w" event={"ID":"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c","Type":"ContainerDied","Data":"20f14a5373b34053f0620d65ddcd08cbfb3615366d1dde2f1195a064f4363d4d"}
Oct 30 22:34:12.248618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:12.248583    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"]
Oct 30 22:34:12.290122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:12.290082    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"]
Oct 30 22:34:13.270959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.270918    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:13.358146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.358104    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-kube-api-access-kpvhx\" (UniqueName: \"kubernetes.io/projected/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-kube-api-access-kpvhx\") pod \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") "
Oct 30 22:34:13.358416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.358173    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-xks-ca\") pod \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") "
Oct 30 22:34:13.358489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.358461    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-openshift-ca\") pod \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\" (UID: \"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c\") "
Oct 30 22:34:13.359538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.359497    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-openshift-ca" pod "1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c" (UID: "1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:13.362320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.362288    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-kube-api-access-kpvhx" pod "1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c" (UID: "1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c"). InnerVolumeSpecName "kube-api-access-kpvhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:13.362526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.362494    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-xks-ca" pod "1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c" (UID: "1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:13.461508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.461468    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:13.461508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.461508    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:13.461790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.461524    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvhx\" (UniqueName: \"kubernetes.io/projected/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c-kube-api-access-kpvhx\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:13.839514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.839478    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"
Oct 30 22:34:13.839514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.839500    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w" event={"ID":"1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c","Type":"ContainerDied","Data":"7bc4acd4ab834e65c747f4baf39bb9fc093fd97f08c75c2d0fab3b09bd9a20c6"}
Oct 30 22:34:13.839793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:13.839572    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc4acd4ab834e65c747f4baf39bb9fc093fd97f08c75c2d0fab3b09bd9a20c6"
Oct 30 22:34:14.762921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.762874    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"]
Oct 30 22:34:14.763978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.763953    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c" containerName="revoke-keys"
Oct 30 22:34:14.765640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.764131    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c" containerName="revoke-keys"
Oct 30 22:34:14.765640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.764211    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy"
Oct 30 22:34:14.765640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.764238    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c8d384-92f1-40f3-a233-84519c2ee72f" containerName="istio-proxy"
Oct 30 22:34:14.765640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.764601    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c" containerName="revoke-keys"
Oct 30 22:34:14.769137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.769108    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.781647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.781616    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6922d3-openshift-default-dockercfg-5ldsg\""
Oct 30 22:34:14.788751 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.788712    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.788920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.788806    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.788920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.788851    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.789029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.788917    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.789029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.789011    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.789136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.789036    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fhkb\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-kube-api-access-8fhkb\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.789136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.789095    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ac86a2-3a1f-43d8-b834-717d0415646e-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.789278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.789192    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.789340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.789314    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.817734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.801110    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"]
Oct 30 22:34:14.890918 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.890874    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.890918 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.890917    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fhkb\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-kube-api-access-8fhkb\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.891191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.890980    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ac86a2-3a1f-43d8-b834-717d0415646e-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.891191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.891070    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.891191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.891127    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.891415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.891286    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.891415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.891369    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.891415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.891410    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.891573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.891483    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.892564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.892229    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.892564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.892490    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.892564 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.892520    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.892827 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.892573    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ac86a2-3a1f-43d8-b834-717d0415646e-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.892827 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.892741    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.897258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.894429    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.897258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.894909    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.908807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.908776    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:14.908974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:14.908870    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fhkb\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-kube-api-access-8fhkb\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:15.109992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:15.109956    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:15.457187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:15.456833    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"]
Oct 30 22:34:15.464523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:15.464475    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:15.464668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:15.464576    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:15.464668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:15.464620    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:15.900366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:15.900317    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" event={"ID":"c0ac86a2-3a1f-43d8-b834-717d0415646e","Type":"ContainerStarted","Data":"8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814"}
Oct 30 22:34:15.901465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:15.901440    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" event={"ID":"c0ac86a2-3a1f-43d8-b834-717d0415646e","Type":"ContainerStarted","Data":"2f7e874fd0541b0f91e11bcaed3fc527a035bd27dc8996a4b72af5367b9a00ab"}
Oct 30 22:34:15.966554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:15.966479    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podStartSLOduration=1.9664569950000002 podStartE2EDuration="1.966456995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:15.942268216 +0000 UTC m=+1639.981207103" watchObservedRunningTime="2026-10-30 22:34:15.966456995 +0000 UTC m=+1640.005395878"
Oct 30 22:34:16.110377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:16.110348    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:16.111992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:16.111968    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 22:34:16.112137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:16.112023    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 22:34:17.129019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:17.128982    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 22:34:17.129775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:17.129732    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 22:34:17.249704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:17.249657    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9" podUID="850c90a6-d048-460f-92b4-bb81e0ea69c5" containerName="payload-processing" containerID="cri-o://7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166" gracePeriod=25
Oct 30 22:34:17.306318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:17.301125    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42" podUID="213b7946-13df-4673-97bd-e8fe3fde5d80" containerName="payload-pre-processing" containerID="cri-o://54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66" gracePeriod=25
Oct 30 22:34:17.939194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:17.939159    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:34:18.021307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.021196    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850c90a6-d048-460f-92b4-bb81e0ea69c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c90a6-d048-460f-92b4-bb81e0ea69c5-plugins-config-volume\") pod \"850c90a6-d048-460f-92b4-bb81e0ea69c5\" (UID: \"850c90a6-d048-460f-92b4-bb81e0ea69c5\") "
Oct 30 22:34:18.021755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.021724    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850c90a6-d048-460f-92b4-bb81e0ea69c5-kube-api-access-8vmjg\" (UniqueName: \"kubernetes.io/projected/850c90a6-d048-460f-92b4-bb81e0ea69c5-kube-api-access-8vmjg\") pod \"850c90a6-d048-460f-92b4-bb81e0ea69c5\" (UID: \"850c90a6-d048-460f-92b4-bb81e0ea69c5\") "
Oct 30 22:34:18.023412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.023381    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850c90a6-d048-460f-92b4-bb81e0ea69c5-plugins-config-volume" pod "850c90a6-d048-460f-92b4-bb81e0ea69c5" (UID: "850c90a6-d048-460f-92b4-bb81e0ea69c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:18.114308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.110694    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 22:34:18.114308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.110759    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 22:34:18.141252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.131696    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c90a6-d048-460f-92b4-bb81e0ea69c5-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:18.173359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.173330    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:34:18.225834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.225794    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850c90a6-d048-460f-92b4-bb81e0ea69c5-kube-api-access-8vmjg" pod "850c90a6-d048-460f-92b4-bb81e0ea69c5" (UID: "850c90a6-d048-460f-92b4-bb81e0ea69c5"). InnerVolumeSpecName "kube-api-access-8vmjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:18.232486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.231602    4581 generic.go:356] "Generic (PLEG): container finished" podID="850c90a6-d048-460f-92b4-bb81e0ea69c5" containerID="7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166" exitCode=0
Oct 30 22:34:18.232486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.231713    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"
Oct 30 22:34:18.232943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.232910    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9" event={"ID":"850c90a6-d048-460f-92b4-bb81e0ea69c5","Type":"ContainerDied","Data":"7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166"}
Oct 30 22:34:18.233041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.232979    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9" event={"ID":"850c90a6-d048-460f-92b4-bb81e0ea69c5","Type":"ContainerDied","Data":"ec1dae9c2e80ac7c8847afffe7dc6b211763832b56e10094ef8efebfd7e7118a"}
Oct 30 22:34:18.233041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.233009    4581 scope.go:122] "RemoveContainer" containerID="7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166"
Oct 30 22:34:18.234635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.234604    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213b7946-13df-4673-97bd-e8fe3fde5d80-kube-api-access-rrj5w\" (UniqueName: \"kubernetes.io/projected/213b7946-13df-4673-97bd-e8fe3fde5d80-kube-api-access-rrj5w\") pod \"213b7946-13df-4673-97bd-e8fe3fde5d80\" (UID: \"213b7946-13df-4673-97bd-e8fe3fde5d80\") "
Oct 30 22:34:18.234891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.234872    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213b7946-13df-4673-97bd-e8fe3fde5d80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213b7946-13df-4673-97bd-e8fe3fde5d80-plugins-config-volume\") pod \"213b7946-13df-4673-97bd-e8fe3fde5d80\" (UID: \"213b7946-13df-4673-97bd-e8fe3fde5d80\") "
Oct 30 22:34:18.236257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.236149    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmjg\" (UniqueName: \"kubernetes.io/projected/850c90a6-d048-460f-92b4-bb81e0ea69c5-kube-api-access-8vmjg\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:18.239032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.238999    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213b7946-13df-4673-97bd-e8fe3fde5d80-plugins-config-volume" pod "213b7946-13df-4673-97bd-e8fe3fde5d80" (UID: "213b7946-13df-4673-97bd-e8fe3fde5d80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:18.243720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.243509    4581 generic.go:356] "Generic (PLEG): container finished" podID="213b7946-13df-4673-97bd-e8fe3fde5d80" containerID="54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66" exitCode=0
Oct 30 22:34:18.245033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.244831    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"
Oct 30 22:34:18.245033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.244921    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42" event={"ID":"213b7946-13df-4673-97bd-e8fe3fde5d80","Type":"ContainerDied","Data":"54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66"}
Oct 30 22:34:18.245033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.244973    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42" event={"ID":"213b7946-13df-4673-97bd-e8fe3fde5d80","Type":"ContainerDied","Data":"04ffe653a278ce9fa0fa96de6f59fb4573bda5c2bb99cb98b275a08638a48949"}
Oct 30 22:34:18.247596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.247569    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213b7946-13df-4673-97bd-e8fe3fde5d80-kube-api-access-rrj5w" pod "213b7946-13df-4673-97bd-e8fe3fde5d80" (UID: "213b7946-13df-4673-97bd-e8fe3fde5d80"). InnerVolumeSpecName "kube-api-access-rrj5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:18.325636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.325559    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"]
Oct 30 22:34:18.326044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.326024    4581 scope.go:122] "RemoveContainer" containerID="7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166"
Oct 30 22:34:18.326996 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:34:18.326958    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166\": container with ID starting with 7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166 not found: ID does not exist" containerID="7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166"
Oct 30 22:34:18.327133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.326996    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166"} err="failed to get container status \"7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166\": rpc error: code = NotFound desc = could not find container \"7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166\": container with ID starting with 7b6b6429d59c94f0c52a11fd460e2be843f22de1cda3b8c0627fa83577d46166 not found: ID does not exist"
Oct 30 22:34:18.327133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.327025    4581 scope.go:122] "RemoveContainer" containerID="54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66"
Oct 30 22:34:18.332905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.332875    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d38f6-5874cbccb9-4r8w9"]
Oct 30 22:34:18.339559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.339534    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrj5w\" (UniqueName: \"kubernetes.io/projected/213b7946-13df-4673-97bd-e8fe3fde5d80-kube-api-access-rrj5w\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:18.339663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.339571    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213b7946-13df-4673-97bd-e8fe3fde5d80-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:18.343280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.343262    4581 scope.go:122] "RemoveContainer" containerID="54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66"
Oct 30 22:34:18.343617 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:34:18.343600    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66\": container with ID starting with 54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66 not found: ID does not exist" containerID="54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66"
Oct 30 22:34:18.343869 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.343844    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66"} err="failed to get container status \"54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66\": rpc error: code = NotFound desc = could not find container \"54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66\": container with ID starting with 54b205a2aab07798bbe0a315b4b9de933555a565d7cae270f3456f22c2406b66 not found: ID does not exist"
Oct 30 22:34:18.701244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.697437    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"]
Oct 30 22:34:18.727245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:18.720069    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d38f6-8fdb5988c-rjl42"]
Oct 30 22:34:19.110394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:19.110347    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 22:34:19.112483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:19.110417    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 22:34:19.299303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:19.298029    4581 generic.go:356] "Generic (PLEG): container finished" podID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerID="7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8" exitCode=0
Oct 30 22:34:19.299303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:19.298092    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" event={"ID":"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4","Type":"ContainerDied","Data":"7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8"}
Oct 30 22:34:19.299303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:19.298596    4581 scope.go:122] "RemoveContainer" containerID="7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8"
Oct 30 22:34:19.305868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:19.305819    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:19.306056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:19.305907    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:19.306056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:19.305940    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:20.127484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:20.122006    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 22:34:20.127484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:20.122068    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 22:34:20.142064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:20.142028    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213b7946-13df-4673-97bd-e8fe3fde5d80" path="/var/lib/kubelet/pods/213b7946-13df-4673-97bd-e8fe3fde5d80/volumes"
Oct 30 22:34:20.142598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:20.142582    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850c90a6-d048-460f-92b4-bb81e0ea69c5" path="/var/lib/kubelet/pods/850c90a6-d048-460f-92b4-bb81e0ea69c5/volumes"
Oct 30 22:34:20.331609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:20.331580    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" event={"ID":"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4","Type":"ContainerStarted","Data":"69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8"}
Oct 30 22:34:20.744474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:20.744437    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:34:20.744699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:20.744607    4581 patch_prober.go:29] interesting pod/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:20.744699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:20.744668    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:34:21.122280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:21.115332    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 22:34:21.122280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:21.115395    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 22:34:21.747147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:21.745064    4581 patch_prober.go:29] interesting pod/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:21.747147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:21.745113    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:34:22.112348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:22.112311    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 22:34:22.136780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:22.136738    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 22:34:22.745667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:22.745631    4581 patch_prober.go:29] interesting pod/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:22.745891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:22.745696    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:34:23.131329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.130611    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:23.131813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.131747    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:23.133929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.133892    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:34:23.234436 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.234405    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"]
Oct 30 22:34:23.235158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.235144    4581 cpu_manager.go:414] "containerMap: removing container" podUID="213b7946-13df-4673-97bd-e8fe3fde5d80" containerName="payload-pre-processing"
Oct 30 22:34:23.235231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.235160    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213b7946-13df-4673-97bd-e8fe3fde5d80" containerName="payload-pre-processing"
Oct 30 22:34:23.235231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.235177    4581 cpu_manager.go:414] "containerMap: removing container" podUID="850c90a6-d048-460f-92b4-bb81e0ea69c5" containerName="payload-processing"
Oct 30 22:34:23.235231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.235183    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850c90a6-d048-460f-92b4-bb81e0ea69c5" containerName="payload-processing"
Oct 30 22:34:23.235358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.235348    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="213b7946-13df-4673-97bd-e8fe3fde5d80" containerName="payload-pre-processing"
Oct 30 22:34:23.235394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.235363    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="850c90a6-d048-460f-92b4-bb81e0ea69c5" containerName="payload-processing"
Oct 30 22:34:23.240275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.240197    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:23.269290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.260494    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f9d87b\""
Oct 30 22:34:23.269290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.260628    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f9d87b-dockercfg-l8nmq\""
Oct 30 22:34:23.337243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.335787    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27120312-1468-480d-9899-8768fd4593d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs\" (UID: \"27120312-1468-480d-9899-8768fd4593d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:23.337243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.335894    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrs6z\" (UniqueName: \"kubernetes.io/projected/27120312-1468-480d-9899-8768fd4593d3-kube-api-access-hrs6z\") pod \"payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs\" (UID: \"27120312-1468-480d-9899-8768fd4593d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:23.393359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.381288    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"]
Oct 30 22:34:23.447245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.444361    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27120312-1468-480d-9899-8768fd4593d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs\" (UID: \"27120312-1468-480d-9899-8768fd4593d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:23.447245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.444481    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrs6z\" (UniqueName: \"kubernetes.io/projected/27120312-1468-480d-9899-8768fd4593d3-kube-api-access-hrs6z\") pod \"payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs\" (UID: \"27120312-1468-480d-9899-8768fd4593d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:23.447245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.446002    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27120312-1468-480d-9899-8768fd4593d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs\" (UID: \"27120312-1468-480d-9899-8768fd4593d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:23.595242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.594970    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrs6z\" (UniqueName: \"kubernetes.io/projected/27120312-1468-480d-9899-8768fd4593d3-kube-api-access-hrs6z\") pod \"payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs\" (UID: \"27120312-1468-480d-9899-8768fd4593d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:23.618280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.609497    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:23.939565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.937850    4581 patch_prober.go:29] interesting pod/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:23.939565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:23.937921    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:34:25.043978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:25.043891    4581 patch_prober.go:29] interesting pod/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:25.044569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:25.044536    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:34:25.079244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:25.075272    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"]
Oct 30 22:34:25.451525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:25.449281    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"]
Oct 30 22:34:25.549007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:25.548968    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de31dd-x9gq7"]
Oct 30 22:34:25.744832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:25.744797    4581 patch_prober.go:29] interesting pod/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:25.745076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:25.745053    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:34:26.082145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:26.081561    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs" event={"ID":"27120312-1468-480d-9899-8768fd4593d3","Type":"ContainerStarted","Data":"2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e"}
Oct 30 22:34:26.082145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:26.081675    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs" event={"ID":"27120312-1468-480d-9899-8768fd4593d3","Type":"ContainerStarted","Data":"334219a7c247959a08b502eb12bb63bd06025aa095b3c573ed719587a529c130"}
Oct 30 22:34:26.082145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:26.081930    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:26.100580 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:26.100548    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306" path="/var/lib/kubelet/pods/915f0b7f-1740-4c1d-8bd7-4cbb7c7f7306/volumes"
Oct 30 22:34:26.119922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:26.119876    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs" podStartSLOduration=3.119827206 podStartE2EDuration="3.119827206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:26.118654956 +0000 UTC m=+1650.157593843" watchObservedRunningTime="2026-10-30 22:34:26.119827206 +0000 UTC m=+1650.158766110"
Oct 30 22:34:26.742971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:26.742928    4581 patch_prober.go:29] interesting pod/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:26.743184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:26.742993    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:34:27.091368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:27.091270    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:34:27.748977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:27.748946    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:34:28.104481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:28.104449    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:34:30.467479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:30.467427    4581 patch_prober.go:29] interesting pod/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:30.478805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:30.468118    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:31.136435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:31.136399    4581 generic.go:356] "Generic (PLEG): container finished" podID="31025e23-34e0-4096-88a1-85cd62392554" containerID="0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040" exitCode=0
Oct 30 22:34:31.136639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:31.136466    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" event={"ID":"31025e23-34e0-4096-88a1-85cd62392554","Type":"ContainerDied","Data":"0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040"}
Oct 30 22:34:31.136902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:31.136889    4581 scope.go:122] "RemoveContainer" containerID="0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040"
Oct 30 22:34:31.138977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:31.138945    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:31.139077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:31.139020    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:31.139077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:31.139054    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:34:32.148205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:32.148174    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" event={"ID":"31025e23-34e0-4096-88a1-85cd62392554","Type":"ContainerStarted","Data":"5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486"}
Oct 30 22:34:32.339160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:32.339125    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:34:32.339433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:32.339412    4581 patch_prober.go:29] interesting pod/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:34:32.339510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:32.339465    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 22:34:33.339713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.339675    4581 patch_prober.go:29] interesting pod/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:34:33.340140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.339731    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 22:34:33.446936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.446903    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"]
Oct 30 22:34:33.454533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.454506    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.484253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.478544    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"]
Oct 30 22:34:33.484667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.481633    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:33.484814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.481712    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:33.484900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.481807    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:33.484981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.481900    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:34:33.511587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.511556    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.511760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.511593    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56g6w\" (UniqueName: \"kubernetes.io/projected/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-kube-api-access-56g6w\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.511805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.511760    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.613048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.613019    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.613275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.613057    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56g6w\" (UniqueName: \"kubernetes.io/projected/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-kube-api-access-56g6w\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.613275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.613147    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.613889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.613862    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.615793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.615771    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.630811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.630784    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56g6w\" (UniqueName: \"kubernetes.io/projected/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-kube-api-access-56g6w\") pod \"maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:33.795831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:33.795800    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:34.206959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:34.206932    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"]
Oct 30 22:34:34.208571 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:34:34.208536    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac5ac40_2cc2_4e2e_8f5f_640b92f8739f.slice/crio-d0ffcf9b7f9369e5f4926d016087085b1b30dac7c4b8cb5f922b490840a017fa WatchSource:0}: Error finding container d0ffcf9b7f9369e5f4926d016087085b1b30dac7c4b8cb5f922b490840a017fa: Status 404 returned error can't find the container with id d0ffcf9b7f9369e5f4926d016087085b1b30dac7c4b8cb5f922b490840a017fa
Oct 30 22:34:34.338978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:34.338935    4581 patch_prober.go:29] interesting pod/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:34:34.339159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:34.339001    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 22:34:35.205360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:35.205305    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7" event={"ID":"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f","Type":"ContainerStarted","Data":"1c327b244f844bb66c729d586daf4880cb3e1ed6ca73da0bf40fd3bd47e7b4ce"}
Oct 30 22:34:35.205776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:35.205409    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7" event={"ID":"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f","Type":"ContainerStarted","Data":"d0ffcf9b7f9369e5f4926d016087085b1b30dac7c4b8cb5f922b490840a017fa"}
Oct 30 22:34:35.244637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:35.244576    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7" podStartSLOduration=2.244559133 podStartE2EDuration="2.244559133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:35.239275412 +0000 UTC m=+1659.278214300" watchObservedRunningTime="2026-10-30 22:34:35.244559133 +0000 UTC m=+1659.283498019"
Oct 30 22:34:35.339427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:35.339389    4581 patch_prober.go:29] interesting pod/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:34:35.339620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:35.339476    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 22:34:36.216668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:36.216580    4581 generic.go:356] "Generic (PLEG): container finished" podID="7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f" containerID="1c327b244f844bb66c729d586daf4880cb3e1ed6ca73da0bf40fd3bd47e7b4ce" exitCode=0
Oct 30 22:34:36.217134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:36.216700    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7" event={"ID":"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f","Type":"ContainerDied","Data":"1c327b244f844bb66c729d586daf4880cb3e1ed6ca73da0bf40fd3bd47e7b4ce"}
Oct 30 22:34:36.339192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:36.339153    4581 patch_prober.go:29] interesting pod/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:34:36.339395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:36.339260    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 22:34:37.097076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.097044    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:34:37.341363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.340821    4581 patch_prober.go:29] interesting pod/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:34:37.341363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.340887    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 22:34:37.525903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.525878    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:37.609792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.609754    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-kube-api-access-56g6w\" (UniqueName: \"kubernetes.io/projected/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-kube-api-access-56g6w\") pod \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") "
Oct 30 22:34:37.610027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.610006    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-xks-ca\") pod \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") "
Oct 30 22:34:37.610109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.610048    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-openshift-ca\") pod \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\" (UID: \"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f\") "
Oct 30 22:34:37.610942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.610915    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-openshift-ca" pod "7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f" (UID: "7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:37.613117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.613078    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-xks-ca" pod "7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f" (UID: "7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:37.616566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.616540    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-kube-api-access-56g6w" pod "7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f" (UID: "7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f"). InnerVolumeSpecName "kube-api-access-56g6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:37.712730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.712691    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56g6w\" (UniqueName: \"kubernetes.io/projected/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-kube-api-access-56g6w\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:37.712730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.712730    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:37.713008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:37.712748    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:34:38.243480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:38.243447    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"
Oct 30 22:34:38.243480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:38.243466    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7" event={"ID":"7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f","Type":"ContainerDied","Data":"d0ffcf9b7f9369e5f4926d016087085b1b30dac7c4b8cb5f922b490840a017fa"}
Oct 30 22:34:38.243749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:38.243524    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ffcf9b7f9369e5f4926d016087085b1b30dac7c4b8cb5f922b490840a017fa"
Oct 30 22:34:38.345163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:38.345136    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:34:38.345676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:38.345573    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:34:38.346335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:38.346311    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:34:40.819462 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:34:40.819423    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:34:59.331643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.331615    4581 scope.go:122] "RemoveContainer" containerID="a3f7eba88a0abc01994e5ab1de47e6f6474075aaeef29b52c917a11ffde9b2fd"
Oct 30 22:34:59.343917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.343897    4581 scope.go:122] "RemoveContainer" containerID="a850e889d47fabae82c76c34da19e446ff76087c63374bbd759bc359c14f7344"
Oct 30 22:34:59.879714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.878965    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"]
Oct 30 22:34:59.881679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.881477    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f" containerName="revoke-keys"
Oct 30 22:34:59.881679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.881561    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f" containerName="revoke-keys"
Oct 30 22:34:59.883759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.882950    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f" containerName="revoke-keys"
Oct 30 22:34:59.897350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.897316    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.905459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.905428    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"]
Oct 30 22:34:59.948537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.948495    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2q82\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-kube-api-access-c2q82\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.948756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.948583    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.948756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.948668    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.948756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.948740    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.948934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.948812    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.949017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.948993    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef3b1d-a680-4780-ac73-4b41f033d942-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.949167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.949032    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.949167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.949123    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:34:59.949330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:34:59.949205    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef3b1d-a680-4780-ac73-4b41f033d942-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051081    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051189    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef3b1d-a680-4780-ac73-4b41f033d942-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051266    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051346    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051448    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef3b1d-a680-4780-ac73-4b41f033d942-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051518    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2q82\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-kube-api-access-c2q82\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051553    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051591    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051650    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051725    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.051992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.051968    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.052590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.052244    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.052590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.052322    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef3b1d-a680-4780-ac73-4b41f033d942-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.052590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.052547    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.053907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.053884    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.054454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.054433    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef3b1d-a680-4780-ac73-4b41f033d942-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.069405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.069370    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.069604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.069579    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2q82\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-kube-api-access-c2q82\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.216470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.216367    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:00.549351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.549329    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"]
Oct 30 22:35:00.550517 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:35:00.550487    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaef3b1d_a680_4780_ac73_4b41f033d942.slice/crio-a75d4a0dc4384d75cc9e00e98bf2e812b8ceecf8cf9ae14645bc4bffdd900b65 WatchSource:0}: Error finding container a75d4a0dc4384d75cc9e00e98bf2e812b8ceecf8cf9ae14645bc4bffdd900b65: Status 404 returned error can't find the container with id a75d4a0dc4384d75cc9e00e98bf2e812b8ceecf8cf9ae14645bc4bffdd900b65
Oct 30 22:35:00.555518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.555477    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:00.555604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.555569    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:00.555647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:00.555612    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:01.556243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:01.554690    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" event={"ID":"caef3b1d-a680-4780-ac73-4b41f033d942","Type":"ContainerStarted","Data":"a411e06106481a6badcef8a3da6bd000d01923e245685dde15024a1e747ac076"}
Oct 30 22:35:01.556243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:01.554757    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" event={"ID":"caef3b1d-a680-4780-ac73-4b41f033d942","Type":"ContainerStarted","Data":"a75d4a0dc4384d75cc9e00e98bf2e812b8ceecf8cf9ae14645bc4bffdd900b65"}
Oct 30 22:35:01.709292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:01.707549    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podStartSLOduration=2.707527888 podStartE2EDuration="2.707527888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:01.652036645 +0000 UTC m=+1685.690975537" watchObservedRunningTime="2026-10-30 22:35:01.707527888 +0000 UTC m=+1685.746466781"
Oct 30 22:35:02.216876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:02.216832    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:02.218284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:02.218260    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:35:02.218415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:02.218314    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:35:02.692835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:02.692779    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"]
Oct 30 22:35:02.712799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:02.712608    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-30edd7-qfdnb"]
Oct 30 22:35:03.216908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:03.216872    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:35:03.217101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:03.216931    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:35:04.100493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:04.100457    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a811bf0-3bdb-4b3b-9bd5-0100dfef137a" path="/var/lib/kubelet/pods/0a811bf0-3bdb-4b3b-9bd5-0100dfef137a/volumes"
Oct 30 22:35:04.217209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:04.217178    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:35:04.217432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:04.217271    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:35:05.217453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:05.217409    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:35:05.217983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:05.217474    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:35:06.223666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:06.223629    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:35:06.224338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:06.224304    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:35:07.217660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:07.217626    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:35:07.217845 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:07.217679    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:35:08.223142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:08.223100    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:08.223724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:08.223508    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:08.224279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:08.224212    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:35:10.647677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:10.647643    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"]
Oct 30 22:35:10.648187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:10.647884    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy" containerID="cri-o://c71979a8e7771f8501175b03430e90e8cab573792dc43ee980f7a42dbb0162ef" gracePeriod=30
Oct 30 22:35:10.711814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:10.711781    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"]
Oct 30 22:35:10.719368 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:10.719339    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf9c2f-x489g"]
Oct 30 22:35:10.923296 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:35:10.919641    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:35:11.318842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.318762    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"]
Oct 30 22:35:11.325256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.325234    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.338506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.338480    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:35:11.347931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.347894    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:35:11.349117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.349098    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:35:11.349261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.349145    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:35:11.352839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.352383    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.352839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.352515    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnhk\" (UniqueName: \"kubernetes.io/projected/0bbb4a24-c840-4a60-9c3f-195994e62dc6-kube-api-access-slnhk\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.352839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.352554    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.367801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.367770    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"]
Oct 30 22:35:11.454236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.454182    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.454440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.454359    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnhk\" (UniqueName: \"kubernetes.io/projected/0bbb4a24-c840-4a60-9c3f-195994e62dc6-kube-api-access-slnhk\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.454440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.454398    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.455053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.455028    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.457263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.457212    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.472235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.472193    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnhk\" (UniqueName: \"kubernetes.io/projected/0bbb4a24-c840-4a60-9c3f-195994e62dc6-kube-api-access-slnhk\") pod \"maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.641655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.641625    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:11.961296 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:35:11.961257    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbb4a24_c840_4a60_9c3f_195994e62dc6.slice/crio-3a076dee8d0a309f37c5e9471c0fb8abc48f0cf2475708065d4b82cb0d071c7f WatchSource:0}: Error finding container 3a076dee8d0a309f37c5e9471c0fb8abc48f0cf2475708065d4b82cb0d071c7f: Status 404 returned error can't find the container with id 3a076dee8d0a309f37c5e9471c0fb8abc48f0cf2475708065d4b82cb0d071c7f
Oct 30 22:35:11.961811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:11.961738    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"]
Oct 30 22:35:12.100238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:12.100193    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5389fc3-5275-4114-8a2b-61d99b0f53bc" path="/var/lib/kubelet/pods/a5389fc3-5275-4114-8a2b-61d99b0f53bc/volumes"
Oct 30 22:35:12.744750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:12.744652    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq" event={"ID":"0bbb4a24-c840-4a60-9c3f-195994e62dc6","Type":"ContainerStarted","Data":"30e907f65dbe8cbfa6ea6576cd93be83e6e7b71931df9cd804c6e97a33a24164"}
Oct 30 22:35:12.744750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:12.744723    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq" event={"ID":"0bbb4a24-c840-4a60-9c3f-195994e62dc6","Type":"ContainerStarted","Data":"3a076dee8d0a309f37c5e9471c0fb8abc48f0cf2475708065d4b82cb0d071c7f"}
Oct 30 22:35:12.787045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:12.786980    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq" podStartSLOduration=1.786960676 podStartE2EDuration="1.786960676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:12.781101642 +0000 UTC m=+1696.820040529" watchObservedRunningTime="2026-10-30 22:35:12.786960676 +0000 UTC m=+1696.825899563"
Oct 30 22:35:13.759668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:13.759632    4581 generic.go:356] "Generic (PLEG): container finished" podID="0bbb4a24-c840-4a60-9c3f-195994e62dc6" containerID="30e907f65dbe8cbfa6ea6576cd93be83e6e7b71931df9cd804c6e97a33a24164" exitCode=0
Oct 30 22:35:13.760108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:13.759714    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq" event={"ID":"0bbb4a24-c840-4a60-9c3f-195994e62dc6","Type":"ContainerDied","Data":"30e907f65dbe8cbfa6ea6576cd93be83e6e7b71931df9cd804c6e97a33a24164"}
Oct 30 22:35:15.470765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.470730    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:15.593291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.593259    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-openshift-ca\") pod \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") "
Oct 30 22:35:15.593653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.593632    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbb4a24-c840-4a60-9c3f-195994e62dc6-kube-api-access-slnhk\" (UniqueName: \"kubernetes.io/projected/0bbb4a24-c840-4a60-9c3f-195994e62dc6-kube-api-access-slnhk\") pod \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") "
Oct 30 22:35:15.593705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.593652    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-openshift-ca" pod "0bbb4a24-c840-4a60-9c3f-195994e62dc6" (UID: "0bbb4a24-c840-4a60-9c3f-195994e62dc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:15.593705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.593697    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-xks-ca\") pod \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\" (UID: \"0bbb4a24-c840-4a60-9c3f-195994e62dc6\") "
Oct 30 22:35:15.594387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.594345    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:15.596615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.596586    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-xks-ca" pod "0bbb4a24-c840-4a60-9c3f-195994e62dc6" (UID: "0bbb4a24-c840-4a60-9c3f-195994e62dc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:15.598373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.598348    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbb4a24-c840-4a60-9c3f-195994e62dc6-kube-api-access-slnhk" pod "0bbb4a24-c840-4a60-9c3f-195994e62dc6" (UID: "0bbb4a24-c840-4a60-9c3f-195994e62dc6"). InnerVolumeSpecName "kube-api-access-slnhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:15.695856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.695832    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slnhk\" (UniqueName: \"kubernetes.io/projected/0bbb4a24-c840-4a60-9c3f-195994e62dc6-kube-api-access-slnhk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:15.695964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.695859    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbb4a24-c840-4a60-9c3f-195994e62dc6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:15.803413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.803380    4581 generic.go:356] "Generic (PLEG): container finished" podID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerID="c71979a8e7771f8501175b03430e90e8cab573792dc43ee980f7a42dbb0162ef" exitCode=0
Oct 30 22:35:15.803567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.803452    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" event={"ID":"6511e97e-31ed-4652-b7a1-e0661ea7c88e","Type":"ContainerDied","Data":"c71979a8e7771f8501175b03430e90e8cab573792dc43ee980f7a42dbb0162ef"}
Oct 30 22:35:15.803567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.803513    4581 scope.go:122] "RemoveContainer" containerID="9d9f02ca54d4c40c8b19ef4d06279b7cc8dd9b53c890b17c0f3aa19c6d09cb55"
Oct 30 22:35:15.805234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.805199    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"
Oct 30 22:35:15.805359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.805197    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq" event={"ID":"0bbb4a24-c840-4a60-9c3f-195994e62dc6","Type":"ContainerDied","Data":"3a076dee8d0a309f37c5e9471c0fb8abc48f0cf2475708065d4b82cb0d071c7f"}
Oct 30 22:35:15.805359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:15.805350    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a076dee8d0a309f37c5e9471c0fb8abc48f0cf2475708065d4b82cb0d071c7f"
Oct 30 22:35:16.028085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.028059    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:35:16.103331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103295    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-certs\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.103530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103400    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-podinfo\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.103530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103461    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-envoy\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.103654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103558    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-data\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.103654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103602    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-socket\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.103757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103717    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istiod-ca-cert\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.103757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103752    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-kube-api-access-zfsfq\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-kube-api-access-zfsfq\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.103857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103794    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-credential-socket\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.103971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.103952    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-token\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-token\") pod \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\" (UID: \"6511e97e-31ed-4652-b7a1-e0661ea7c88e\") "
Oct 30 22:35:16.105634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.105600    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-data" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:16.106132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.105938    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-certs" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:16.108007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.106507    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-credential-socket" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:16.108007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.107846    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-token" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:16.108293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.108192    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-socket" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:16.108538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.108521    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istiod-ca-cert" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:16.111047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.111017    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-kube-api-access-zfsfq" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "kube-api-access-zfsfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:16.111400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.111371    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-envoy" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:16.116253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.115409    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-podinfo" pod "6511e97e-31ed-4652-b7a1-e0661ea7c88e" (UID: "6511e97e-31ed-4652-b7a1-e0661ea7c88e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:35:16.206693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206651    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.206693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206684    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsfq\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-kube-api-access-zfsfq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.206693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206698    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.206959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206711    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.206959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206723    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.206959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206734    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.206959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206744    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.206959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206758    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.206959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.206769    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511e97e-31ed-4652-b7a1-e0661ea7c88e-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:16.817619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.817530    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"
Oct 30 22:35:16.817619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.817542    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2" event={"ID":"6511e97e-31ed-4652-b7a1-e0661ea7c88e","Type":"ContainerDied","Data":"609d23e7952a1483098b11d786d2f098ee3f2e981553c5a0509dcb9f13b338f1"}
Oct 30 22:35:16.818134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.817638    4581 scope.go:122] "RemoveContainer" containerID="c71979a8e7771f8501175b03430e90e8cab573792dc43ee980f7a42dbb0162ef"
Oct 30 22:35:16.891242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.891193    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"]
Oct 30 22:35:16.900887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:16.900852    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-30edd7-openshift-default-69b49fb475-6hzw2"]
Oct 30 22:35:18.098519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:18.098489    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" path="/var/lib/kubelet/pods/6511e97e-31ed-4652-b7a1-e0661ea7c88e/volumes"
Oct 30 22:35:18.774881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:18.774848    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"]
Oct 30 22:35:18.775378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:18.775339    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy" containerID="cri-o://5548a25208e2e07fa81676dd8ceb0eaa6acc9dcfaf07afd74906300dcac66571" gracePeriod=30
Oct 30 22:35:20.924188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.924017    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"]
Oct 30 22:35:20.925162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925135    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy"
Oct 30 22:35:20.925162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925163    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy"
Oct 30 22:35:20.925437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925243    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy"
Oct 30 22:35:20.925437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925253    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy"
Oct 30 22:35:20.925437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925286    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0bbb4a24-c840-4a60-9c3f-195994e62dc6" containerName="revoke-keys"
Oct 30 22:35:20.925437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925295    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbb4a24-c840-4a60-9c3f-195994e62dc6" containerName="revoke-keys"
Oct 30 22:35:20.925695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925464    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy"
Oct 30 22:35:20.925695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925486    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6511e97e-31ed-4652-b7a1-e0661ea7c88e" containerName="istio-proxy"
Oct 30 22:35:20.925695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.925498    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbb4a24-c840-4a60-9c3f-195994e62dc6" containerName="revoke-keys"
Oct 30 22:35:20.931360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.931333    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:20.970045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.963094    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"]
Oct 30 22:35:20.970045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:20.963392    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6013c8-openshift-default-dockercfg-xj9fh\""
Oct 30 22:35:21.006734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.006691    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-data\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.007041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.006821    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-token\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.007041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.006949    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-credential-socket\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.007041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.006992    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-socket\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.007277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.007107    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-envoy\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.007277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.007167    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-podinfo\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.007277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.007203    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlvz\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-kube-api-access-sxlvz\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.007460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.007277    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istiod-ca-cert\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.007460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.007307    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-certs\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117299    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-token\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117392    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-credential-socket\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117443    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-socket\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117571    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-envoy\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117634    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-podinfo\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117673    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlvz\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-kube-api-access-sxlvz\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117718    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istiod-ca-cert\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117749    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-certs\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.117867    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-data\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.120250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.118428    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-data\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.132295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.129731    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-credential-socket\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.159249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.156759    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-certs\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.159249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.157769    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istiod-ca-cert\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.159249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.158045    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-socket\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.296849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.291626    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-podinfo\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.309244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.305499    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-envoy\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.313242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.310378    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlvz\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-kube-api-access-sxlvz\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.317241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.316891    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-token\") pod \"e2e-inf-b-6013c8-openshift-default-96f465855-llgxc\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") " pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.574924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.574817    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:21.845850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.845740    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"]
Oct 30 22:35:21.887842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.887808    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"]
Oct 30 22:35:21.940610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.940571    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"]
Oct 30 22:35:21.941053 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:35:21.940644    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636c8afd_b441_43f0_b5fb_ddc2a33dd2c9.slice/crio-ca123f88759d85f55fb31283aa10ebb4edeecd979a9dde175a5491c1f5bc81f5 WatchSource:0}: Error finding container ca123f88759d85f55fb31283aa10ebb4edeecd979a9dde175a5491c1f5bc81f5: Status 404 returned error can't find the container with id ca123f88759d85f55fb31283aa10ebb4edeecd979a9dde175a5491c1f5bc81f5
Oct 30 22:35:21.943796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.943763    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:21.943875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.943832    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:21.943875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:21.943864    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:22.928273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:22.928229    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" event={"ID":"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9","Type":"ContainerStarted","Data":"305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469"}
Oct 30 22:35:22.928458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:22.928297    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" event={"ID":"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9","Type":"ContainerStarted","Data":"ca123f88759d85f55fb31283aa10ebb4edeecd979a9dde175a5491c1f5bc81f5"}
Oct 30 22:35:22.979533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:22.979469    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podStartSLOduration=2.97944933 podStartE2EDuration="2.97944933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:22.974342267 +0000 UTC m=+1707.013281153" watchObservedRunningTime="2026-10-30 22:35:22.97944933 +0000 UTC m=+1707.018388220"
Oct 30 22:35:23.576325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:23.576285    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:23.583039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:23.583007    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:23.954718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:23.954679    4581 generic.go:356] "Generic (PLEG): container finished" podID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerID="5548a25208e2e07fa81676dd8ceb0eaa6acc9dcfaf07afd74906300dcac66571" exitCode=0
Oct 30 22:35:23.956490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:23.956455    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" event={"ID":"dfb3f78a-618e-484e-b9ba-49ee3dd6c589","Type":"ContainerDied","Data":"5548a25208e2e07fa81676dd8ceb0eaa6acc9dcfaf07afd74906300dcac66571"}
Oct 30 22:35:23.956788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:23.956744    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:23.956917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:23.956798    4581 scope.go:122] "RemoveContainer" containerID="cdeac9cb3e79b8a74f179f3761b0a685825c6a5d4f00418a2d5ad6676e8e9648"
Oct 30 22:35:23.962502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:23.962476    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:35:24.118697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.118521    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:35:24.196184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196098    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-certs\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.196184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196151    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-podinfo\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.196184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196184    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-credential-socket\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.196518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196291    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-data\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.196518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196348    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-kube-api-access-fl5rc\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-kube-api-access-fl5rc\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.196518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196388    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istiod-ca-cert\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.196518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196418    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-certs" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:24.196518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196518    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-envoy\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.197809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196566    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-token\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-token\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.197809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.196709    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-socket\") pod \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\" (UID: \"dfb3f78a-618e-484e-b9ba-49ee3dd6c589\") "
Oct 30 22:35:24.197809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.197747    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.201422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.201377    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-data" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:24.209592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.202441    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-credential-socket" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:24.214577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.214197    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istiod-ca-cert" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:24.216866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.216832    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-token" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:24.233014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.232422    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-podinfo" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:35:24.234285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.234238    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-socket" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:24.236157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.236130    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-kube-api-access-fl5rc" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "kube-api-access-fl5rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:24.237689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.237666    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-envoy" pod "dfb3f78a-618e-484e-b9ba-49ee3dd6c589" (UID: "dfb3f78a-618e-484e-b9ba-49ee3dd6c589"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:24.299126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.299093    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.299126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.299121    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.299126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.299133    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.299433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.299142    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl5rc\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-kube-api-access-fl5rc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.299433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.299152    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.299433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.299162    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.299433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.299169    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.299433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.299177    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb3f78a-618e-484e-b9ba-49ee3dd6c589-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:24.966387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.966342    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"
Oct 30 22:35:24.966627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.966406    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b" event={"ID":"dfb3f78a-618e-484e-b9ba-49ee3dd6c589","Type":"ContainerDied","Data":"cc6a362cfd5b18fcefd4ee58ada5cc64da7dd9dfc791ef830fd7eeae9d18d9f2"}
Oct 30 22:35:24.966627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:24.966530    4581 scope.go:122] "RemoveContainer" containerID="5548a25208e2e07fa81676dd8ceb0eaa6acc9dcfaf07afd74906300dcac66571"
Oct 30 22:35:25.026556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:25.026519    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"]
Oct 30 22:35:25.031590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:25.031560    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bf9c2f-openshift-default-7c859cb8dc-d962b"]
Oct 30 22:35:26.098826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.098792    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" path="/var/lib/kubelet/pods/dfb3f78a-618e-484e-b9ba-49ee3dd6c589/volumes"
Oct 30 22:35:26.407074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.407039    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"]
Oct 30 22:35:26.407868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.407851    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy"
Oct 30 22:35:26.407933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.407869    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy"
Oct 30 22:35:26.407933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.407879    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy"
Oct 30 22:35:26.407933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.407884    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy"
Oct 30 22:35:26.408028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.408012    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy"
Oct 30 22:35:26.408028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.408020    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb3f78a-618e-484e-b9ba-49ee3dd6c589" containerName="istio-proxy"
Oct 30 22:35:26.413404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.413378    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:26.435035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.434999    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"]
Oct 30 22:35:26.438264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.438239    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-999470-dockercfg-fqwvf\""
Oct 30 22:35:26.501309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.501272    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-999470\""
Oct 30 22:35:26.566173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.566132    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjz6c\" (UniqueName: \"kubernetes.io/projected/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-kube-api-access-jjz6c\") pod \"payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm\" (UID: \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:26.668599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.668491    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm\" (UID: \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:26.815186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.815137    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjz6c\" (UniqueName: \"kubernetes.io/projected/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-kube-api-access-jjz6c\") pod \"payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm\" (UID: \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:26.815825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.815802    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm\" (UID: \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:26.816862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.816811    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm\" (UID: \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:26.819128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.819106    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"]
Oct 30 22:35:26.826483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.824924    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:26.845262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.843143    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"]
Oct 30 22:35:26.850290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.850086    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjz6c\" (UniqueName: \"kubernetes.io/projected/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-kube-api-access-jjz6c\") pod \"payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm\" (UID: \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:26.850491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.850283    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8" podUID="08e653cb-8b10-4f5a-98ee-0e04fbdd5e61" containerName="payload-processing" containerID="cri-o://cb43fd2cc3adb9ec5e4411594930f55b8678d693a90d5b333d27bf92cf3607bc" gracePeriod=25
Oct 30 22:35:26.888911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.888875    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk" podUID="4bdfa881-7aec-4cec-81cc-e15c0efc41a2" containerName="payload-pre-processing" containerID="cri-o://bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379" gracePeriod=25
Oct 30 22:35:26.989934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.989897    4581 generic.go:356] "Generic (PLEG): container finished" podID="08e653cb-8b10-4f5a-98ee-0e04fbdd5e61" containerID="cb43fd2cc3adb9ec5e4411594930f55b8678d693a90d5b333d27bf92cf3607bc" exitCode=0
Oct 30 22:35:26.990112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:26.989965    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8" event={"ID":"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61","Type":"ContainerDied","Data":"cb43fd2cc3adb9ec5e4411594930f55b8678d693a90d5b333d27bf92cf3607bc"}
Oct 30 22:35:27.020502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.020469    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-999470-99979b95c-2x956\" (UID: \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:27.020668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.020638    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2d8\" (UniqueName: \"kubernetes.io/projected/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-kube-api-access-vh2d8\") pod \"payload-processing-e2e-inf-a-999470-99979b95c-2x956\" (UID: \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:27.027991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.027962    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:27.132461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.130005    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-999470-99979b95c-2x956\" (UID: \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:27.132461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.130210    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2d8\" (UniqueName: \"kubernetes.io/projected/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-kube-api-access-vh2d8\") pod \"payload-processing-e2e-inf-a-999470-99979b95c-2x956\" (UID: \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:27.132461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.131837    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-999470-99979b95c-2x956\" (UID: \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:27.157996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.153904    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2d8\" (UniqueName: \"kubernetes.io/projected/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-kube-api-access-vh2d8\") pod \"payload-processing-e2e-inf-a-999470-99979b95c-2x956\" (UID: \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:27.203345 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.203308    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:35:27.263287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.261898    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:35:27.317430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.317395    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:27.333157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.333120    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-plugins-config-volume\") pod \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\" (UID: \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\") "
Oct 30 22:35:27.333351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.333194    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-kube-api-access-zvrsf\" (UniqueName: \"kubernetes.io/projected/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-kube-api-access-zvrsf\") pod \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\" (UID: \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\") "
Oct 30 22:35:27.333351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.333315    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-kube-api-access-xdhfj\" (UniqueName: \"kubernetes.io/projected/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-kube-api-access-xdhfj\") pod \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\" (UID: \"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61\") "
Oct 30 22:35:27.334386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.334350    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-plugins-config-volume" pod "08e653cb-8b10-4f5a-98ee-0e04fbdd5e61" (UID: "08e653cb-8b10-4f5a-98ee-0e04fbdd5e61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:27.350830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.350782    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-kube-api-access-zvrsf" pod "4bdfa881-7aec-4cec-81cc-e15c0efc41a2" (UID: "4bdfa881-7aec-4cec-81cc-e15c0efc41a2"). InnerVolumeSpecName "kube-api-access-zvrsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:27.351927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.351895    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:27.352060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.351934    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrsf\" (UniqueName: \"kubernetes.io/projected/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-kube-api-access-zvrsf\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:27.371377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.371330    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-kube-api-access-xdhfj" pod "08e653cb-8b10-4f5a-98ee-0e04fbdd5e61" (UID: "08e653cb-8b10-4f5a-98ee-0e04fbdd5e61"). InnerVolumeSpecName "kube-api-access-xdhfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:27.409315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.408975    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"]
Oct 30 22:35:27.452736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.452705    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-plugins-config-volume\") pod \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\" (UID: \"4bdfa881-7aec-4cec-81cc-e15c0efc41a2\") "
Oct 30 22:35:27.454117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.454083    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdhfj\" (UniqueName: \"kubernetes.io/projected/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61-kube-api-access-xdhfj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:27.454538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.454507    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-plugins-config-volume" pod "4bdfa881-7aec-4cec-81cc-e15c0efc41a2" (UID: "4bdfa881-7aec-4cec-81cc-e15c0efc41a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:27.555611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.555571    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdfa881-7aec-4cec-81cc-e15c0efc41a2-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:27.662306 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:35:27.662274    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c99d1f_cfcc_4c80_a316_aa96b27aeafc.slice/crio-626f146bd41d994541d6e09c218b27c0341d5aaab75ad6d5590571862cd4d1ac WatchSource:0}: Error finding container 626f146bd41d994541d6e09c218b27c0341d5aaab75ad6d5590571862cd4d1ac: Status 404 returned error can't find the container with id 626f146bd41d994541d6e09c218b27c0341d5aaab75ad6d5590571862cd4d1ac
Oct 30 22:35:27.662454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:27.662355    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"]
Oct 30 22:35:28.001586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.001501    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"
Oct 30 22:35:28.001586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.001515    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8" event={"ID":"08e653cb-8b10-4f5a-98ee-0e04fbdd5e61","Type":"ContainerDied","Data":"fc4dbdcca83485c42f34723b66d18003147777872aaf716b6f114c787565e9cd"}
Oct 30 22:35:28.001841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.001592    4581 scope.go:122] "RemoveContainer" containerID="cb43fd2cc3adb9ec5e4411594930f55b8678d693a90d5b333d27bf92cf3607bc"
Oct 30 22:35:28.004171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.004141    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956" event={"ID":"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc","Type":"ContainerStarted","Data":"76bce2d210285b28c7e1d8d81c95971983ae433c8fd24abb9e57178d0111592d"}
Oct 30 22:35:28.004373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.004200    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956" event={"ID":"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc","Type":"ContainerStarted","Data":"626f146bd41d994541d6e09c218b27c0341d5aaab75ad6d5590571862cd4d1ac"}
Oct 30 22:35:28.004373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.004367    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:28.005805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.005777    4581 generic.go:356] "Generic (PLEG): container finished" podID="4bdfa881-7aec-4cec-81cc-e15c0efc41a2" containerID="bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379" exitCode=0
Oct 30 22:35:28.005942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.005837    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"
Oct 30 22:35:28.005942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.005854    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk" event={"ID":"4bdfa881-7aec-4cec-81cc-e15c0efc41a2","Type":"ContainerDied","Data":"bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379"}
Oct 30 22:35:28.005942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.005909    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk" event={"ID":"4bdfa881-7aec-4cec-81cc-e15c0efc41a2","Type":"ContainerDied","Data":"e472a840b917abc25ffae1165653f38f38425b0309401871c3daee4e0e1c4ee1"}
Oct 30 22:35:28.007990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.007968    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm" event={"ID":"df98ade7-2aa1-4bea-b2f2-6901e7564e5a","Type":"ContainerStarted","Data":"a4414354e6b1a3c46a0c873dd66faee563c7e7dfdc1dcfa317a1c868b6178ed0"}
Oct 30 22:35:28.008093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.008016    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm" event={"ID":"df98ade7-2aa1-4bea-b2f2-6901e7564e5a","Type":"ContainerStarted","Data":"5099fb27cf9b100c3b5e260227031036c94bb52940cde906d7cb0bb74bd0e944"}
Oct 30 22:35:28.008093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.008071    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:28.015931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.015912    4581 scope.go:122] "RemoveContainer" containerID="bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379"
Oct 30 22:35:28.031200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.031180    4581 scope.go:122] "RemoveContainer" containerID="bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379"
Oct 30 22:35:28.031554 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:35:28.031535    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379\": container with ID starting with bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379 not found: ID does not exist" containerID="bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379"
Oct 30 22:35:28.031622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.031563    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379"} err="failed to get container status \"bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379\": rpc error: code = NotFound desc = could not find container \"bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379\": container with ID starting with bb2daf20eac6d73bffc6f0d20ad6850305f5ef0d189ec8338dbc1b1482630379 not found: ID does not exist"
Oct 30 22:35:28.062230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.062152    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm" podStartSLOduration=2.062132778 podStartE2EDuration="2.062132778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:28.057308819 +0000 UTC m=+1712.096247705" watchObservedRunningTime="2026-10-30 22:35:28.062132778 +0000 UTC m=+1712.101071665"
Oct 30 22:35:28.095613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.095554    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956" podStartSLOduration=2.095535486 podStartE2EDuration="2.095535486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:28.090729012 +0000 UTC m=+1712.129667900" watchObservedRunningTime="2026-10-30 22:35:28.095535486 +0000 UTC m=+1712.134474373"
Oct 30 22:35:28.128816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.128779    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"]
Oct 30 22:35:28.137964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.137931    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f7439-55564f8b9f-xttgk"]
Oct 30 22:35:28.166754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.166712    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"]
Oct 30 22:35:28.171979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.171948    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f7439-db569fdbc-lx8l8"]
Oct 30 22:35:28.296277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.296153    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"]
Oct 30 22:35:28.297001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.296982    4581 cpu_manager.go:414] "containerMap: removing container" podUID="08e653cb-8b10-4f5a-98ee-0e04fbdd5e61" containerName="payload-processing"
Oct 30 22:35:28.297097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.297003    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e653cb-8b10-4f5a-98ee-0e04fbdd5e61" containerName="payload-processing"
Oct 30 22:35:28.297097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.297044    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4bdfa881-7aec-4cec-81cc-e15c0efc41a2" containerName="payload-pre-processing"
Oct 30 22:35:28.297097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.297049    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdfa881-7aec-4cec-81cc-e15c0efc41a2" containerName="payload-pre-processing"
Oct 30 22:35:28.297199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.297155    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdfa881-7aec-4cec-81cc-e15c0efc41a2" containerName="payload-pre-processing"
Oct 30 22:35:28.297199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.297168    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e653cb-8b10-4f5a-98ee-0e04fbdd5e61" containerName="payload-processing"
Oct 30 22:35:28.302501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.302478    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:28.314940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.314906    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6013c8\""
Oct 30 22:35:28.315069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.314958    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6013c8-dockercfg-4qldc\""
Oct 30 22:35:28.320642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.320617    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"]
Oct 30 22:35:28.504715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.431524    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899aba0-a556-4a34-a0db-d6205a977d62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw\" (UID: \"0899aba0-a556-4a34-a0db-d6205a977d62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:28.504715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.431709    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5czb\" (UniqueName: \"kubernetes.io/projected/0899aba0-a556-4a34-a0db-d6205a977d62-kube-api-access-q5czb\") pod \"payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw\" (UID: \"0899aba0-a556-4a34-a0db-d6205a977d62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:28.547372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.547284    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899aba0-a556-4a34-a0db-d6205a977d62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw\" (UID: \"0899aba0-a556-4a34-a0db-d6205a977d62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:28.547709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.547687    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5czb\" (UniqueName: \"kubernetes.io/projected/0899aba0-a556-4a34-a0db-d6205a977d62-kube-api-access-q5czb\") pod \"payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw\" (UID: \"0899aba0-a556-4a34-a0db-d6205a977d62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:28.570738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.570703    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899aba0-a556-4a34-a0db-d6205a977d62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw\" (UID: \"0899aba0-a556-4a34-a0db-d6205a977d62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:28.694280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.689576    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5czb\" (UniqueName: \"kubernetes.io/projected/0899aba0-a556-4a34-a0db-d6205a977d62-kube-api-access-q5czb\") pod \"payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw\" (UID: \"0899aba0-a556-4a34-a0db-d6205a977d62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:28.697234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.697182    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"]
Oct 30 22:35:28.707069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.705887    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:28.714699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.711993    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"]
Oct 30 22:35:28.884629 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.884589    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbrsh\" (UniqueName: \"kubernetes.io/projected/3f20acdb-7d98-459b-b668-91f1d4784470-kube-api-access-sbrsh\") pod \"payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg\" (UID: \"3f20acdb-7d98-459b-b668-91f1d4784470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:28.885242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.885197    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f20acdb-7d98-459b-b668-91f1d4784470-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg\" (UID: \"3f20acdb-7d98-459b-b668-91f1d4784470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:28.918700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.918669    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:28.988830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.988787    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbrsh\" (UniqueName: \"kubernetes.io/projected/3f20acdb-7d98-459b-b668-91f1d4784470-kube-api-access-sbrsh\") pod \"payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg\" (UID: \"3f20acdb-7d98-459b-b668-91f1d4784470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:28.988999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.988983    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f20acdb-7d98-459b-b668-91f1d4784470-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg\" (UID: \"3f20acdb-7d98-459b-b668-91f1d4784470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:28.989777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:28.989755    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f20acdb-7d98-459b-b668-91f1d4784470-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg\" (UID: \"3f20acdb-7d98-459b-b668-91f1d4784470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:29.006044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:29.006004    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbrsh\" (UniqueName: \"kubernetes.io/projected/3f20acdb-7d98-459b-b668-91f1d4784470-kube-api-access-sbrsh\") pod \"payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg\" (UID: \"3f20acdb-7d98-459b-b668-91f1d4784470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:29.033020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:29.032894    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:29.231776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:29.231745    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"]
Oct 30 22:35:29.232982 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:35:29.232958    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0899aba0_a556_4a34_a0db_d6205a977d62.slice/crio-b6d2bb241246f66df8fa4ff6da39e142ea80eaf1483c5fc8501ea43a64504a05 WatchSource:0}: Error finding container b6d2bb241246f66df8fa4ff6da39e142ea80eaf1483c5fc8501ea43a64504a05: Status 404 returned error can't find the container with id b6d2bb241246f66df8fa4ff6da39e142ea80eaf1483c5fc8501ea43a64504a05
Oct 30 22:35:29.362965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:29.362936    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"]
Oct 30 22:35:30.046115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.046083    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg" event={"ID":"3f20acdb-7d98-459b-b668-91f1d4784470","Type":"ContainerStarted","Data":"4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445"}
Oct 30 22:35:30.046419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.046143    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg" event={"ID":"3f20acdb-7d98-459b-b668-91f1d4784470","Type":"ContainerStarted","Data":"58fa39368fa8b23acee77505e76502a7b904ba62f1c5b0b1ed248e2909d76ff4"}
Oct 30 22:35:30.046419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.046172    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:30.048009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.047985    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw" event={"ID":"0899aba0-a556-4a34-a0db-d6205a977d62","Type":"ContainerStarted","Data":"4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f"}
Oct 30 22:35:30.048133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.048023    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw" event={"ID":"0899aba0-a556-4a34-a0db-d6205a977d62","Type":"ContainerStarted","Data":"b6d2bb241246f66df8fa4ff6da39e142ea80eaf1483c5fc8501ea43a64504a05"}
Oct 30 22:35:30.048133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.048092    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:30.080631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.080582    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg" podStartSLOduration=2.080567135 podStartE2EDuration="2.080567135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:30.079542081 +0000 UTC m=+1714.118480966" watchObservedRunningTime="2026-10-30 22:35:30.080567135 +0000 UTC m=+1714.119506037"
Oct 30 22:35:30.098824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.098786    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e653cb-8b10-4f5a-98ee-0e04fbdd5e61" path="/var/lib/kubelet/pods/08e653cb-8b10-4f5a-98ee-0e04fbdd5e61/volumes"
Oct 30 22:35:30.099426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.099400    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdfa881-7aec-4cec-81cc-e15c0efc41a2" path="/var/lib/kubelet/pods/4bdfa881-7aec-4cec-81cc-e15c0efc41a2/volumes"
Oct 30 22:35:30.114501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.114148    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw" podStartSLOduration=2.114132105 podStartE2EDuration="2.114132105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:30.112109849 +0000 UTC m=+1714.151048734" watchObservedRunningTime="2026-10-30 22:35:30.114132105 +0000 UTC m=+1714.153070991"
Oct 30 22:35:30.806099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.805493    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"]
Oct 30 22:35:30.817201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:30.817149    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a1ee7-ckg6w"]
Oct 30 22:35:32.112522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:32.112486    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c" path="/var/lib/kubelet/pods/1c7c13bc-107b-4b8f-913d-c55b6bd2ea0c/volumes"
Oct 30 22:35:36.977994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:36.977956    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"]
Oct 30 22:35:36.989516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:36.989484    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d38f6-gqgw6"]
Oct 30 22:35:38.100851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:38.100816    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf619604-275f-4d75-af9a-009ea1ca0926" path="/var/lib/kubelet/pods/bf619604-275f-4d75-af9a-009ea1ca0926/volumes"
Oct 30 22:35:38.993982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:38.993940    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"]
Oct 30 22:35:38.994276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:38.994205    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy" containerID="cri-o://5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486" gracePeriod=30
Oct 30 22:35:39.040319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.040285    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:35:39.040903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.040879    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:35:39.600029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.599989    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"]
Oct 30 22:35:39.606526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.606501    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.623010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.622976    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"]
Oct 30 22:35:39.626333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.626310    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:35:39.626509 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.626309    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:35:39.626509 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.626373    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:35:39.626643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.626426    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:35:39.760863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.760825    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nhrl\" (UniqueName: \"kubernetes.io/projected/66e7d2dd-34ac-482d-a185-d5453c068c85-kube-api-access-2nhrl\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.761068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.760989    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.761068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.761027    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.863497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.863466    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhrl\" (UniqueName: \"kubernetes.io/projected/66e7d2dd-34ac-482d-a185-d5453c068c85-kube-api-access-2nhrl\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.863693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.863570    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.863693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.863599    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.864452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.864422    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.866295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.866254    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.881354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.881328    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nhrl\" (UniqueName: \"kubernetes.io/projected/66e7d2dd-34ac-482d-a185-d5453c068c85-kube-api-access-2nhrl\") pod \"maas-api-revoke-keys-e2e-api-b-62236a-cz646\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:39.928819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:39.928786    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:40.251045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:40.250934    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"]
Oct 30 22:35:40.254593 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:35:40.254563    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e7d2dd_34ac_482d_a185_d5453c068c85.slice/crio-ea3eba81e8b172df1fd6a3fe738b98132f28dd5b2f4a4fb925cde7298b6e31bd WatchSource:0}: Error finding container ea3eba81e8b172df1fd6a3fe738b98132f28dd5b2f4a4fb925cde7298b6e31bd: Status 404 returned error can't find the container with id ea3eba81e8b172df1fd6a3fe738b98132f28dd5b2f4a4fb925cde7298b6e31bd
Oct 30 22:35:40.440823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:40.440787    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646" event={"ID":"66e7d2dd-34ac-482d-a185-d5453c068c85","Type":"ContainerStarted","Data":"ea3eba81e8b172df1fd6a3fe738b98132f28dd5b2f4a4fb925cde7298b6e31bd"}
Oct 30 22:35:40.818432 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:35:40.818394    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:35:41.063365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:41.063285    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:35:41.063527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:41.063420    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:35:41.452307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:41.452240    4581 generic.go:356] "Generic (PLEG): container finished" podID="66e7d2dd-34ac-482d-a185-d5453c068c85" containerID="dc416bd79dabee4318d27d83f2435b3bca0e98ecd70195ffa8a814edc3bdbd86" exitCode=0
Oct 30 22:35:41.452486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:41.452331    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646" event={"ID":"66e7d2dd-34ac-482d-a185-d5453c068c85","Type":"ContainerDied","Data":"dc416bd79dabee4318d27d83f2435b3bca0e98ecd70195ffa8a814edc3bdbd86"}
Oct 30 22:35:42.667920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.667894    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:42.824656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.824552    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-xks-ca\") pod \"66e7d2dd-34ac-482d-a185-d5453c068c85\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") "
Oct 30 22:35:42.824815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.824767    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e7d2dd-34ac-482d-a185-d5453c068c85-kube-api-access-2nhrl\" (UniqueName: \"kubernetes.io/projected/66e7d2dd-34ac-482d-a185-d5453c068c85-kube-api-access-2nhrl\") pod \"66e7d2dd-34ac-482d-a185-d5453c068c85\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") "
Oct 30 22:35:42.824931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.824914    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-openshift-ca\") pod \"66e7d2dd-34ac-482d-a185-d5453c068c85\" (UID: \"66e7d2dd-34ac-482d-a185-d5453c068c85\") "
Oct 30 22:35:42.830682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.827291    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-xks-ca" pod "66e7d2dd-34ac-482d-a185-d5453c068c85" (UID: "66e7d2dd-34ac-482d-a185-d5453c068c85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:42.830682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.827831    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e7d2dd-34ac-482d-a185-d5453c068c85-kube-api-access-2nhrl" pod "66e7d2dd-34ac-482d-a185-d5453c068c85" (UID: "66e7d2dd-34ac-482d-a185-d5453c068c85"). InnerVolumeSpecName "kube-api-access-2nhrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:42.830682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.828418    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-openshift-ca" pod "66e7d2dd-34ac-482d-a185-d5453c068c85" (UID: "66e7d2dd-34ac-482d-a185-d5453c068c85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:42.927866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.927826    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:42.927866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.927863    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nhrl\" (UniqueName: \"kubernetes.io/projected/66e7d2dd-34ac-482d-a185-d5453c068c85-kube-api-access-2nhrl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:42.928151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:42.927880    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e7d2dd-34ac-482d-a185-d5453c068c85-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:43.479771 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:43.479733    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"
Oct 30 22:35:43.479962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:43.479766    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646" event={"ID":"66e7d2dd-34ac-482d-a185-d5453c068c85","Type":"ContainerDied","Data":"ea3eba81e8b172df1fd6a3fe738b98132f28dd5b2f4a4fb925cde7298b6e31bd"}
Oct 30 22:35:43.479962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:43.479861    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3eba81e8b172df1fd6a3fe738b98132f28dd5b2f4a4fb925cde7298b6e31bd"
Oct 30 22:35:44.302958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.302923    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:35:44.470953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.470917    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-data\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.471151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.470977    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31025e23-34e0-4096-88a1-85cd62392554-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31025e23-34e0-4096-88a1-85cd62392554-istio-podinfo\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.471151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.471030    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-certs\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.471151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.471049    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-credential-socket\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.471433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.471282    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-data" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:44.471433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.471417    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-certs" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:44.471837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.471579    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-socket" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:44.471837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.471675    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-socket\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.471837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.471825    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-envoy\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.472061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.471970    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-istio-token\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-istio-token\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.472237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.472181    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-kube-api-access-q9nw2\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-kube-api-access-q9nw2\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.472360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.472254    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31025e23-34e0-4096-88a1-85cd62392554-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31025e23-34e0-4096-88a1-85cd62392554-istiod-ca-cert\") pod \"31025e23-34e0-4096-88a1-85cd62392554\" (UID: \"31025e23-34e0-4096-88a1-85cd62392554\") "
Oct 30 22:35:44.473149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.473020    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31025e23-34e0-4096-88a1-85cd62392554-istiod-ca-cert" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:44.473627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.473382    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:44.473627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.473406    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:44.473627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.473421    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31025e23-34e0-4096-88a1-85cd62392554-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:44.473627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.473466    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:44.473627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.473594    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-credential-socket" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:44.474849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.474565    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31025e23-34e0-4096-88a1-85cd62392554-istio-podinfo" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:35:44.475168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.475141    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-kube-api-access-q9nw2" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "kube-api-access-q9nw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:44.475322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.475174    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-envoy" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:44.476272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.476205    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-istio-token" pod "31025e23-34e0-4096-88a1-85cd62392554" (UID: "31025e23-34e0-4096-88a1-85cd62392554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:44.492432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.492404    4581 generic.go:356] "Generic (PLEG): container finished" podID="31025e23-34e0-4096-88a1-85cd62392554" containerID="5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486" exitCode=0
Oct 30 22:35:44.492575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.492484    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"
Oct 30 22:35:44.492575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.492526    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" event={"ID":"31025e23-34e0-4096-88a1-85cd62392554","Type":"ContainerDied","Data":"5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486"}
Oct 30 22:35:44.492675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.492585    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f" event={"ID":"31025e23-34e0-4096-88a1-85cd62392554","Type":"ContainerDied","Data":"19510a77daa263ebf496a86854234dc472a32741f33a3c8515276fe0a7a4f19a"}
Oct 30 22:35:44.492675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.492610    4581 scope.go:122] "RemoveContainer" containerID="5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486"
Oct 30 22:35:44.515847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.515826    4581 scope.go:122] "RemoveContainer" containerID="0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040"
Oct 30 22:35:44.527387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.527358    4581 scope.go:122] "RemoveContainer" containerID="5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486"
Oct 30 22:35:44.527717 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:35:44.527697    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486\": container with ID starting with 5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486 not found: ID does not exist" containerID="5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486"
Oct 30 22:35:44.527772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.527728    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486"} err="failed to get container status \"5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486\": rpc error: code = NotFound desc = could not find container \"5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486\": container with ID starting with 5700b97649af59261a978c82483380bd2d6899dc2f1d1ef47a2ec4adc87a1486 not found: ID does not exist"
Oct 30 22:35:44.527772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.527746    4581 scope.go:122] "RemoveContainer" containerID="0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040"
Oct 30 22:35:44.528007 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:35:44.527987    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040\": container with ID starting with 0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040 not found: ID does not exist" containerID="0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040"
Oct 30 22:35:44.528070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.528018    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040"} err="failed to get container status \"0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040\": rpc error: code = NotFound desc = could not find container \"0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040\": container with ID starting with 0752455a6f1dcf7d5acfa31bb468848ad031590b901cd8c8c77d02d332c38040 not found: ID does not exist"
Oct 30 22:35:44.541195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.541168    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"]
Oct 30 22:35:44.547350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.547326    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7a1ee7-openshift-default-68c5c46ccc-5gx2f"]
Oct 30 22:35:44.574923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.574896    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31025e23-34e0-4096-88a1-85cd62392554-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:44.574923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.574920    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:44.575114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.574934    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31025e23-34e0-4096-88a1-85cd62392554-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:44.575114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.574944    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:44.575114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:44.574953    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9nw2\" (UniqueName: \"kubernetes.io/projected/31025e23-34e0-4096-88a1-85cd62392554-kube-api-access-q9nw2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:35:46.099053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.099016    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31025e23-34e0-4096-88a1-85cd62392554" path="/var/lib/kubelet/pods/31025e23-34e0-4096-88a1-85cd62392554/volumes"
Oct 30 22:35:46.506246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.504572    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"]
Oct 30 22:35:46.506246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.505738    4581 cpu_manager.go:414] "containerMap: removing container" podUID="66e7d2dd-34ac-482d-a185-d5453c068c85" containerName="revoke-keys"
Oct 30 22:35:46.506246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.505761    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e7d2dd-34ac-482d-a185-d5453c068c85" containerName="revoke-keys"
Oct 30 22:35:46.506246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.505800    4581 cpu_manager.go:414] "containerMap: removing container" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy"
Oct 30 22:35:46.506246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.505808    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy"
Oct 30 22:35:46.506246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.506068    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy"
Oct 30 22:35:46.506246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.506091    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e7d2dd-34ac-482d-a185-d5453c068c85" containerName="revoke-keys"
Oct 30 22:35:46.506246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.506108    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy"
Oct 30 22:35:46.602279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.592425    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"]
Oct 30 22:35:46.602279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.592552    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.619256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.616139    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d85da4-openshift-default-dockercfg-z6ttn\""
Oct 30 22:35:46.779963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.779849    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-podinfo\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.779963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.779912    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-credential-socket\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.780337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.779972    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-envoy\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.780337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.780008    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbfh\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-kube-api-access-9mbfh\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.780337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.780133    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-certs\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.780337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.780172    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istiod-ca-cert\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.780337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.780205    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-token\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.780337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.780320    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-socket\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.780806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.780374    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-data\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.885468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.883288    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-envoy\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.885778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.885754    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbfh\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-kube-api-access-9mbfh\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.886096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.886075    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-certs\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.886251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.886234    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istiod-ca-cert\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.886534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.886507    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-certs\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.886623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.886581    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-token\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.886692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.886674    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-socket\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.886755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.886724    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-data\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.886956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.886939    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-podinfo\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.887015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.886992    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-credential-socket\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.888043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.887427    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-credential-socket\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.888043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.887537    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istiod-ca-cert\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.888043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.887821    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-data\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.888299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.888085    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-socket\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.891066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.891045    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-envoy\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.891802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.891783    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-podinfo\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.903581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.903552    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbfh\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-kube-api-access-9mbfh\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:46.903581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:46.903577    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-token\") pod \"e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") " pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:47.119584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:47.119556    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:47.457489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:47.457452    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"]
Oct 30 22:35:47.464914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:47.464867    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:47.465066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:47.465011    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:47.465124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:47.465096    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:35:47.592782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:47.592742    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" event={"ID":"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56","Type":"ContainerStarted","Data":"fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0"}
Oct 30 22:35:47.593041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:47.593013    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" event={"ID":"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56","Type":"ContainerStarted","Data":"0596d3f400dac42b622462a98ba450bc5c4714152b6da77fab9a1acfcf143cb9"}
Oct 30 22:35:47.632486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:47.632430    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" podStartSLOduration=1.632416308 podStartE2EDuration="1.632416308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:47.630729346 +0000 UTC m=+1731.669668231" watchObservedRunningTime="2026-10-30 22:35:47.632416308 +0000 UTC m=+1731.671355193"
Oct 30 22:35:48.119744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:48.119702    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:48.126320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:48.126297    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:48.603360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:48.603331    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:48.604399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:48.604375    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:35:54.942300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:54.942257    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"]
Oct 30 22:35:54.942833 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:54.942518    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy" containerID="cri-o://20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2" gracePeriod=30
Oct 30 22:35:55.213306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:55.202719    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"]
Oct 30 22:35:55.213516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:55.213469    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" containerID="cri-o://bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c" gracePeriod=30
Oct 30 22:35:55.477257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:55.477094    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"]
Oct 30 22:35:55.478517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:55.478204    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy" containerID="cri-o://2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a" gracePeriod=30
Oct 30 22:35:59.414144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:59.414098    4581 scope.go:122] "RemoveContainer" containerID="2040d1cddd09764cc6aea0a36d318684e26ed8f466fc31a183d8ee5fb4c7ac9b"
Oct 30 22:35:59.429487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:35:59.429439    4581 scope.go:122] "RemoveContainer" containerID="eb9bed9b747d26326b23f7cb9e8a59f167671e8dd817fc024ff0b36cf012d1b2"
Oct 30 22:36:00.258694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.258661    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:36:00.285696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.285656    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-kube-api-access-hfmrk\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-kube-api-access-hfmrk\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.285905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.285880    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-certs\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.286020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.285998    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-podinfo\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.286155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286083    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-data\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.286155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286122    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istiod-ca-cert\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.286318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286156    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-socket\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.286318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286190    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-credential-socket\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.286318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286266    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-envoy\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.286449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286379    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-data" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.286519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286488    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-credential-socket" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.286574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286526    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-token\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-token\") pod \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\" (UID: \"3c958d0c-10a1-4b47-a326-e9b877ce1f68\") "
Oct 30 22:36:00.286969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286850    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istiod-ca-cert" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:00.286969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286901    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-socket" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.286969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.286953    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-certs" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.287384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.287364    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.287384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.287382    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.287384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.287391    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.287594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.287403    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.287594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.287412    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.288706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.288681    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-podinfo" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:00.288913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.288898    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-kube-api-access-hfmrk" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "kube-api-access-hfmrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.288997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.288951    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-envoy" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.289150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.289133    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-token" pod "3c958d0c-10a1-4b47-a326-e9b877ce1f68" (UID: "3c958d0c-10a1-4b47-a326-e9b877ce1f68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.389615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.389583    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.389615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.389613    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.389615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.389622    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.389866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.389630    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmrk\" (UniqueName: \"kubernetes.io/projected/3c958d0c-10a1-4b47-a326-e9b877ce1f68-kube-api-access-hfmrk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.608833 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.608802    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:36:00.694932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.694894    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-credential-socket\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.695121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695016    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-socket\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.695121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695095    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-data\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.695275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695133    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-kube-api-access-kf9tr\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-kube-api-access-kf9tr\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.695275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695232    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-credential-socket" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.695275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695245    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae9826f-c2f9-485f-82e3-179c3493ff78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae9826f-c2f9-485f-82e3-179c3493ff78-istiod-ca-cert\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.695440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695321    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-token\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-token\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.695440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695418    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-data" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.695440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695434    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-certs\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.695597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695542    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-envoy\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.695657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695609    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-podinfo\") pod \"eae9826f-c2f9-485f-82e3-179c3493ff78\" (UID: \"eae9826f-c2f9-485f-82e3-179c3493ff78\") "
Oct 30 22:36:00.696489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.696454    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.696489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.696485    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.696686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.695608    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-socket" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.696866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.696840    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-certs" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.698003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.697458    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae9826f-c2f9-485f-82e3-179c3493ff78-istiod-ca-cert" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:00.700739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.700695    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-podinfo" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:00.700846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.700756    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-kube-api-access-kf9tr" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "kube-api-access-kf9tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.701673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.701647    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-token" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.703002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.702967    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-envoy" pod "eae9826f-c2f9-485f-82e3-179c3493ff78" (UID: "eae9826f-c2f9-485f-82e3-179c3493ff78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.797593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.797557    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:36:00.798525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.798503    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.798586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.798528    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.798586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.798537    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.798586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.798545    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae9826f-c2f9-485f-82e3-179c3493ff78-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.798586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.798555    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf9tr\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-kube-api-access-kf9tr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.798586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.798564    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae9826f-c2f9-485f-82e3-179c3493ff78-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.798586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.798572    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae9826f-c2f9-485f-82e3-179c3493ff78-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.901468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901435    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-credential-socket\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.901655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901493    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-podinfo\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.901655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901540    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-kube-api-access-qntzn\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-kube-api-access-qntzn\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.901655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901566    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-data\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.901655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901638    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-socket\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.901900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901676    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istiod-ca-cert\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.901900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901724    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-token\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-token\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.901900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901759    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-certs\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.901900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901769    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-credential-socket" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.901900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901809    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-envoy\") pod \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\" (UID: \"887a1cf8-96c9-444f-a7cd-6847a0c28f74\") "
Oct 30 22:36:00.902272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.901978    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-socket" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.902272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.902178    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-data" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.902554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.902530    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istiod-ca-cert" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:00.902788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.902765    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-certs" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.903537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.903509    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.903659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.903541    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.903659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.903558    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.903659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.903572    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.903659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.903587    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:00.905165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.905103    4581 generic.go:356] "Generic (PLEG): container finished" podID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerID="20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2" exitCode=0
Oct 30 22:36:00.905165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.905160    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"
Oct 30 22:36:00.905494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.905310    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-kube-api-access-qntzn" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "kube-api-access-qntzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.905494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.905321    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-envoy" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.905494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.905268    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" event={"ID":"3c958d0c-10a1-4b47-a326-e9b877ce1f68","Type":"ContainerDied","Data":"20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2"}
Oct 30 22:36:00.905494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.905417    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp" event={"ID":"3c958d0c-10a1-4b47-a326-e9b877ce1f68","Type":"ContainerDied","Data":"ca2430dc6b431467de34e77f81d4c05e9d8aef418886a0620915af912e6a4f18"}
Oct 30 22:36:00.905494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.905448    4581 scope.go:122] "RemoveContainer" containerID="20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2"
Oct 30 22:36:00.905685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.905558    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-podinfo" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:00.906734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.906472    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-token" pod "887a1cf8-96c9-444f-a7cd-6847a0c28f74" (UID: "887a1cf8-96c9-444f-a7cd-6847a0c28f74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.907682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.907653    4581 generic.go:356] "Generic (PLEG): container finished" podID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerID="bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c" exitCode=0
Oct 30 22:36:00.907956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.907940    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"
Oct 30 22:36:00.908538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.908505    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" event={"ID":"eae9826f-c2f9-485f-82e3-179c3493ff78","Type":"ContainerDied","Data":"bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c"}
Oct 30 22:36:00.908654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.908566    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" event={"ID":"eae9826f-c2f9-485f-82e3-179c3493ff78","Type":"ContainerDied","Data":"ee9d3bd696d090443eab60a60f7ee784c728ccd1450a146292c95d3ddb4945df"}
Oct 30 22:36:00.920208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.919987    4581 generic.go:356] "Generic (PLEG): container finished" podID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerID="2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a" exitCode=0
Oct 30 22:36:00.920208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.920057    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" event={"ID":"887a1cf8-96c9-444f-a7cd-6847a0c28f74","Type":"ContainerDied","Data":"2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a"}
Oct 30 22:36:00.920208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.920114    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv" event={"ID":"887a1cf8-96c9-444f-a7cd-6847a0c28f74","Type":"ContainerDied","Data":"cf4822e38064fa3ea2c40beae3522f947aa954c0dd3fe4b234bb863bbd24f602"}
Oct 30 22:36:00.920208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.920154    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"
Oct 30 22:36:00.923876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.923830    4581 scope.go:122] "RemoveContainer" containerID="20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2"
Oct 30 22:36:00.924310 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:36:00.924283    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2\": container with ID starting with 20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2 not found: ID does not exist" containerID="20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2"
Oct 30 22:36:00.924407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.924321    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2"} err="failed to get container status \"20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2\": rpc error: code = NotFound desc = could not find container \"20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2\": container with ID starting with 20d8252e638c63c7c675d2ac435695ec4e9eed515df7457476fbc12bc8015cf2 not found: ID does not exist"
Oct 30 22:36:00.924407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.924348    4581 scope.go:122] "RemoveContainer" containerID="bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c"
Oct 30 22:36:00.939767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.939740    4581 scope.go:122] "RemoveContainer" containerID="bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c"
Oct 30 22:36:00.940110 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:36:00.940087    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c\": container with ID starting with bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c not found: ID does not exist" containerID="bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c"
Oct 30 22:36:00.940162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.940121    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c"} err="failed to get container status \"bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c\": rpc error: code = NotFound desc = could not find container \"bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c\": container with ID starting with bafa812c204ab45764c3a22b4362218f9572c165d26ddbeef291c2595e9b8e9c not found: ID does not exist"
Oct 30 22:36:00.940162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.940139    4581 scope.go:122] "RemoveContainer" containerID="2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a"
Oct 30 22:36:00.953942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.953922    4581 scope.go:122] "RemoveContainer" containerID="2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a"
Oct 30 22:36:00.954343 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:36:00.954310    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a\": container with ID starting with 2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a not found: ID does not exist" containerID="2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a"
Oct 30 22:36:00.954450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.954367    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a"} err="failed to get container status \"2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a\": rpc error: code = NotFound desc = could not find container \"2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a\": container with ID starting with 2879242ffe313359f7e2c0d500ff9d99970ada07f9139efb1136ca874965c92a not found: ID does not exist"
Oct 30 22:36:00.971293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.971263    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"]
Oct 30 22:36:00.977110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:00.977086    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-d5lpp"]
Oct 30 22:36:01.007942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.007289    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.007942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.007324    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntzn\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-kube-api-access-qntzn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.007942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.007341    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.007942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.007356    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887a1cf8-96c9-444f-a7cd-6847a0c28f74-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:01.020787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.020755    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"]
Oct 30 22:36:01.027385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.027351    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-svsgv"]
Oct 30 22:36:01.044812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.044724    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"]
Oct 30 22:36:01.050947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.050918    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn"]
Oct 30 22:36:01.515919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.515880    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:36:01.516090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:01.515938    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-67nxn" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:36:02.097962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:02.097926    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" path="/var/lib/kubelet/pods/3c958d0c-10a1-4b47-a326-e9b877ce1f68/volumes"
Oct 30 22:36:02.098494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:02.098479    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" path="/var/lib/kubelet/pods/887a1cf8-96c9-444f-a7cd-6847a0c28f74/volumes"
Oct 30 22:36:02.098922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:02.098908    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" path="/var/lib/kubelet/pods/eae9826f-c2f9-485f-82e3-179c3493ff78/volumes"
Oct 30 22:36:03.598962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:03.598925    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"]
Oct 30 22:36:03.606427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:03.606387    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b3e43-bmmm7"]
Oct 30 22:36:04.099195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:04.099151    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f" path="/var/lib/kubelet/pods/7ac5ac40-2cc2-4e2e-8f5f-640b92f8739f/volumes"
Oct 30 22:36:09.932357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:09.932254    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"]
Oct 30 22:36:09.932939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:09.932462    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy" containerID="cri-o://7bad5c7ad95b81d1cd7f48b386fa22497d650e9dfa10654b0a021399cc4a86aa" gracePeriod=30
Oct 30 22:36:10.844764 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:36:10.844718    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:36:12.395273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.394641    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"]
Oct 30 22:36:12.395789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.395765    4581 cpu_manager.go:414] "containerMap: removing container" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy"
Oct 30 22:36:12.395858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.395791    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy"
Oct 30 22:36:12.395858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.395825    4581 cpu_manager.go:414] "containerMap: removing container" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy"
Oct 30 22:36:12.395858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.395832    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31025e23-34e0-4096-88a1-85cd62392554" containerName="istio-proxy"
Oct 30 22:36:12.395858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.395847    4581 cpu_manager.go:414] "containerMap: removing container" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy"
Oct 30 22:36:12.395858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.395855    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy"
Oct 30 22:36:12.396112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.395918    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy"
Oct 30 22:36:12.396112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.395925    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy"
Oct 30 22:36:12.396265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.396145    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="887a1cf8-96c9-444f-a7cd-6847a0c28f74" containerName="istio-proxy"
Oct 30 22:36:12.396265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.396167    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae9826f-c2f9-485f-82e3-179c3493ff78" containerName="istio-proxy"
Oct 30 22:36:12.396265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.396185    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c958d0c-10a1-4b47-a326-e9b877ce1f68" containerName="istio-proxy"
Oct 30 22:36:12.402848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.402821    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.415303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.415278    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-40229e-openshift-default-dockercfg-7dwpw\""
Oct 30 22:36:12.433296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.433261    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"]
Oct 30 22:36:12.483809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.483764    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istiod-ca-cert\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.484137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.484109    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.484360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.484151    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-certs\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.484360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.484187    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-data\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.484360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.484249    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbp9c\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-kube-api-access-qbp9c\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.484741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.484454    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-token\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.484741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.484583    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-envoy\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.484741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.484646    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-credential-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.484741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.484720    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-podinfo\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.618775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.618732    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-token\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.623386    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-envoy\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.623455    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-credential-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.623517    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-podinfo\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.623706    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istiod-ca-cert\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.623799    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.623830    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-certs\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.623867    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-data\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.623907    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbp9c\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-kube-api-access-qbp9c\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.633242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.629025    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istiod-ca-cert\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.635159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.634257    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-data\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.635159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.634524    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-credential-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.635159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.634743    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-socket\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.635917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.635875    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-certs\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.653413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.650339    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-token\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.714878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.706131    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbp9c\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-kube-api-access-qbp9c\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.714878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.708282    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-envoy\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.715792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.715727    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-podinfo\") pod \"e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") " pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:12.724450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:12.722570    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:13.229664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:13.229632    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"]
Oct 30 22:36:13.230721 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:36:13.230699    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdd4c93_87a0_45d3_af7d_a5ad7626d81d.slice/crio-a230ce3ff3e220ed147604316fa8b65da239293a76b2a18f6ece3084c1ea9529 WatchSource:0}: Error finding container a230ce3ff3e220ed147604316fa8b65da239293a76b2a18f6ece3084c1ea9529: Status 404 returned error can't find the container with id a230ce3ff3e220ed147604316fa8b65da239293a76b2a18f6ece3084c1ea9529
Oct 30 22:36:13.233619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:13.233587    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:13.233691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:13.233655    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:13.233730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:13.233702    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:14.120256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:14.120178    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" event={"ID":"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d","Type":"ContainerStarted","Data":"6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce"}
Oct 30 22:36:14.120741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:14.120262    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" event={"ID":"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d","Type":"ContainerStarted","Data":"a230ce3ff3e220ed147604316fa8b65da239293a76b2a18f6ece3084c1ea9529"}
Oct 30 22:36:14.173560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:14.173488    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" podStartSLOduration=2.173469022 podStartE2EDuration="2.173469022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:14.166807912 +0000 UTC m=+1758.205746798" watchObservedRunningTime="2026-10-30 22:36:14.173469022 +0000 UTC m=+1758.212407909"
Oct 30 22:36:14.723998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:14.723958    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:14.730560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:14.730532    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:15.139466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.139422    4581 generic.go:356] "Generic (PLEG): container finished" podID="5b191633-9fa0-4662-9221-95b2540c3efc" containerID="7bad5c7ad95b81d1cd7f48b386fa22497d650e9dfa10654b0a021399cc4a86aa" exitCode=0
Oct 30 22:36:15.139986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.139499    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" event={"ID":"5b191633-9fa0-4662-9221-95b2540c3efc","Type":"ContainerDied","Data":"7bad5c7ad95b81d1cd7f48b386fa22497d650e9dfa10654b0a021399cc4a86aa"}
Oct 30 22:36:15.140100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.140076    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:15.142143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.141632    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:36:15.258704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.258676    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:36:15.389810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.389774    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-socket\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.389912    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-kube-api-access-5n8jk\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-kube-api-access-5n8jk\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.389983    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b191633-9fa0-4662-9221-95b2540c3efc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b191633-9fa0-4662-9221-95b2540c3efc-istiod-ca-cert\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390032    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b191633-9fa0-4662-9221-95b2540c3efc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b191633-9fa0-4662-9221-95b2540c3efc-istio-podinfo\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390101    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-socket" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.390253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390193    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-certs\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390252    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-data\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390288    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-envoy\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390358    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-credential-socket\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390405    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-istio-token\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-istio-token\") pod \"5b191633-9fa0-4662-9221-95b2540c3efc\" (UID: \"5b191633-9fa0-4662-9221-95b2540c3efc\") "
Oct 30 22:36:15.390435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390408    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b191633-9fa0-4662-9221-95b2540c3efc-istiod-ca-cert" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:15.390610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390495    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-certs" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.390734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390684    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-data" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.390826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.390704    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-credential-socket" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.391362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.391330    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.391491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.391401    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b191633-9fa0-4662-9221-95b2540c3efc-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.391491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.391418    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.391491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.391431    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.391491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.391445    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.392775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.392745    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-kube-api-access-5n8jk" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "kube-api-access-5n8jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:15.393063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.393036    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b191633-9fa0-4662-9221-95b2540c3efc-istio-podinfo" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:15.393429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.393404    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-envoy" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.393840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.393820    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-istio-token" pod "5b191633-9fa0-4662-9221-95b2540c3efc" (UID: "5b191633-9fa0-4662-9221-95b2540c3efc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:15.515431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.500940    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n8jk\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-kube-api-access-5n8jk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.515431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.500987    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b191633-9fa0-4662-9221-95b2540c3efc-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.515431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.501003    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b191633-9fa0-4662-9221-95b2540c3efc-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.515431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:15.501016    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b191633-9fa0-4662-9221-95b2540c3efc-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.150182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:16.150154    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"
Oct 30 22:36:16.150678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:16.150184    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7" event={"ID":"5b191633-9fa0-4662-9221-95b2540c3efc","Type":"ContainerDied","Data":"6bf3dcfa5f731a02e0fd560652214abbbc5480b80b430f599ae8f75a10cf055e"}
Oct 30 22:36:16.150678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:16.150305    4581 scope.go:122] "RemoveContainer" containerID="7bad5c7ad95b81d1cd7f48b386fa22497d650e9dfa10654b0a021399cc4a86aa"
Oct 30 22:36:16.215685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:16.215648    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"]
Oct 30 22:36:16.220463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:16.220433    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-cnmb7"]
Oct 30 22:36:18.101809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.101764    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" path="/var/lib/kubelet/pods/5b191633-9fa0-4662-9221-95b2540c3efc/volumes"
Oct 30 22:36:18.426969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.426924    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"]
Oct 30 22:36:18.427862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.427836    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy"
Oct 30 22:36:18.427862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.427863    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy"
Oct 30 22:36:18.428116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.428102    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b191633-9fa0-4662-9221-95b2540c3efc" containerName="istio-proxy"
Oct 30 22:36:18.433543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.433514    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:18.455684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.455656    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"]
Oct 30 22:36:18.509591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.509559    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-40229e-dockercfg-2m86v\""
Oct 30 22:36:18.509979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.509926    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-40229e\""
Oct 30 22:36:18.551550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.551510    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea375e9e-122e-486c-9da0-9e731d72db29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps\" (UID: \"ea375e9e-122e-486c-9da0-9e731d72db29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:18.552051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.552025    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjmb\" (UniqueName: \"kubernetes.io/projected/ea375e9e-122e-486c-9da0-9e731d72db29-kube-api-access-tsjmb\") pod \"payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps\" (UID: \"ea375e9e-122e-486c-9da0-9e731d72db29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:18.656426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.656379    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea375e9e-122e-486c-9da0-9e731d72db29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps\" (UID: \"ea375e9e-122e-486c-9da0-9e731d72db29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:18.657964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.657935    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjmb\" (UniqueName: \"kubernetes.io/projected/ea375e9e-122e-486c-9da0-9e731d72db29-kube-api-access-tsjmb\") pod \"payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps\" (UID: \"ea375e9e-122e-486c-9da0-9e731d72db29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:18.658701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.657614    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea375e9e-122e-486c-9da0-9e731d72db29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps\" (UID: \"ea375e9e-122e-486c-9da0-9e731d72db29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:18.763829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.761151    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjmb\" (UniqueName: \"kubernetes.io/projected/ea375e9e-122e-486c-9da0-9e731d72db29-kube-api-access-tsjmb\") pod \"payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps\" (UID: \"ea375e9e-122e-486c-9da0-9e731d72db29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:18.825257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:18.821091    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:19.161920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:19.161884    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"]
Oct 30 22:36:19.162428 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:36:19.162079    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea375e9e_122e_486c_9da0_9e731d72db29.slice/crio-3e8f45582d67e8812527c35948cc59de6bc3760134cc0e1307b47b00533baa20 WatchSource:0}: Error finding container 3e8f45582d67e8812527c35948cc59de6bc3760134cc0e1307b47b00533baa20: Status 404 returned error can't find the container with id 3e8f45582d67e8812527c35948cc59de6bc3760134cc0e1307b47b00533baa20
Oct 30 22:36:19.185675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:19.185643    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps" event={"ID":"ea375e9e-122e-486c-9da0-9e731d72db29","Type":"ContainerStarted","Data":"3e8f45582d67e8812527c35948cc59de6bc3760134cc0e1307b47b00533baa20"}
Oct 30 22:36:20.198015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:20.197967    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps" event={"ID":"ea375e9e-122e-486c-9da0-9e731d72db29","Type":"ContainerStarted","Data":"55c9d9ad647f2d9b27e1c900e6c20021d768357fb0fe852d6fb7803ceae0cd2c"}
Oct 30 22:36:20.198538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:20.198047    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:20.238199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:20.238141    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps" podStartSLOduration=2.238125676 podStartE2EDuration="2.238125676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:20.232693098 +0000 UTC m=+1764.271631984" watchObservedRunningTime="2026-10-30 22:36:20.238125676 +0000 UTC m=+1764.277064562"
Oct 30 22:36:31.217254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:31.216590    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:36:38.537337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:38.537301    4581 generic.go:356] "Generic (PLEG): container finished" podID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerID="305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469" exitCode=0
Oct 30 22:36:38.537337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:38.537334    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" event={"ID":"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9","Type":"ContainerDied","Data":"305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469"}
Oct 30 22:36:38.537865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:38.537822    4581 scope.go:122] "RemoveContainer" containerID="305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469"
Oct 30 22:36:38.539779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:38.539748    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:38.539873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:38.539825    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:38.539873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:38.539857    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:38.957077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:38.957039    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:36:39.548680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:39.548650    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" event={"ID":"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9","Type":"ContainerStarted","Data":"aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209"}
Oct 30 22:36:39.575324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:39.575292    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:36:39.576673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:39.576644    4581 patch_prober.go:29] interesting pod/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:36:39.576838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:39.576703    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 22:36:40.582477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:40.582421    4581 patch_prober.go:29] interesting pod/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:36:40.583089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:40.582503    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 22:36:40.822480 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:36:40.822432    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:36:41.576245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:41.575621    4581 patch_prober.go:29] interesting pod/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:36:41.576245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:41.575707    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 22:36:42.604243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:42.587419    4581 patch_prober.go:29] interesting pod/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:36:42.605366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:42.604844    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 22:36:43.286327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:43.286279    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"]
Oct 30 22:36:43.291939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:43.291902    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f7439-2vdqq"]
Oct 30 22:36:43.577780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:43.577678    4581 patch_prober.go:29] interesting pod/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:36:43.578777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:43.578745    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 22:36:44.103508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:44.103452    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bbb4a24-c840-4a60-9c3f-195994e62dc6" path="/var/lib/kubelet/pods/0bbb4a24-c840-4a60-9c3f-195994e62dc6/volumes"
Oct 30 22:36:44.576786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:44.576685    4581 patch_prober.go:29] interesting pod/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:36:44.577073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:44.577043    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 22:36:45.176126 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.176085    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"]
Oct 30 22:36:45.182239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.182183    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.208598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.202130    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"]
Oct 30 22:36:45.282683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.282645    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.282683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.282694    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss95j\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-kube-api-access-ss95j\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.282978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.282855    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.282978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.282934    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.283059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.283036    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.283103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.283088    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.283253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.283182    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.283325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.283272    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.283398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.283376    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385048    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385105    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss95j\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-kube-api-access-ss95j\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385235    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385319    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385360    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385400    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385468    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385525    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.385604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.385576    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.386425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.386237    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.386590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.386565    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.388014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.387990    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.391566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.391541    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.392566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.392540    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.392848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.392829    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.408559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.408523    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.412813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.412776    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.419058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.419027    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss95j\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-kube-api-access-ss95j\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.510039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.509945    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:45.582864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.582828    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:36:45.584206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.583493    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:36:45.585631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.585573    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:36:45.872068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.872031    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"]
Oct 30 22:36:45.873957 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:36:45.873895    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6cfbab_0eb3_4312_9eb1_9101cf84b148.slice/crio-be5b9e64eb3926a1895f08593de1baa97098786dd5be262c9d75c2eb49d281d3 WatchSource:0}: Error finding container be5b9e64eb3926a1895f08593de1baa97098786dd5be262c9d75c2eb49d281d3: Status 404 returned error can't find the container with id be5b9e64eb3926a1895f08593de1baa97098786dd5be262c9d75c2eb49d281d3
Oct 30 22:36:45.877874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.877837    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:45.877969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.877918    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:45.877969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:45.877951    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:46.703226 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:46.703166    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" event={"ID":"df6cfbab-0eb3-4312-9eb1-9101cf84b148","Type":"ContainerStarted","Data":"a8702974b2d4b7ac99a46754f75cce1a5c409d0fb2d1eb36c7eafa8d2765f04e"}
Oct 30 22:36:46.703649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:46.703257    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" event={"ID":"df6cfbab-0eb3-4312-9eb1-9101cf84b148","Type":"ContainerStarted","Data":"be5b9e64eb3926a1895f08593de1baa97098786dd5be262c9d75c2eb49d281d3"}
Oct 30 22:36:46.758082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:46.758025    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podStartSLOduration=1.758005586 podStartE2EDuration="1.758005586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:46.75150243 +0000 UTC m=+1790.790441318" watchObservedRunningTime="2026-10-30 22:36:46.758005586 +0000 UTC m=+1790.796944531"
Oct 30 22:36:47.511054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:47.511014    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:47.513079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:47.513051    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:36:47.513270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:47.513103    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 22:36:48.511356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:48.511286    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:36:48.511829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:48.511386    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 22:36:49.510601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:49.510564    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:36:49.510803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:49.510641    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 22:36:50.511168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:50.511130    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:36:50.511626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:50.511192    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 22:36:51.522248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:51.514758    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:36:51.522248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:51.514843    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 22:36:52.297333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.295298    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"]
Oct 30 22:36:52.317246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.316800    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.381242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.379589    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bc48cc-openshift-default-dockercfg-h45f9\""
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.530420    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-podinfo\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.530522    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-envoy\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.530666    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-credential-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.530721    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft954\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-kube-api-access-ft954\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.530891    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-token\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.530919    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-certs\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.530959    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-data\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.530989    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istiod-ca-cert\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.531118    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.549717    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 22:36:52.567487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.549789    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 22:36:52.607269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.605259    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"]
Oct 30 22:36:52.632548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632519    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-credential-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.632697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632569    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft954\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-kube-api-access-ft954\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.632766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632705    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-token\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.632766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632739    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-certs\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.632870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632776    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-data\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.632870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632806    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istiod-ca-cert\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.632976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632919    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.632976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632944    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-credential-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.633094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.632998    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-podinfo\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.633094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.633052    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-envoy\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.633211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.633119    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-certs\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.633464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.633436    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-socket\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.633574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.633463    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-data\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.633706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.633664    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istiod-ca-cert\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.635617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.635594    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-envoy\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.635727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.635687    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-podinfo\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.697260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.695273    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-token\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.721920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.721880    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft954\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-kube-api-access-ft954\") pod \"e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") " pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:52.920886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:52.920846    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:53.267208 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:36:53.267157    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c722c1e_996e_4e92_9d8b_0b100e7c8d3f.slice/crio-61660af84f1e67c1bd631095162eeb88e7f04bdc621a6c3109cf2397ed568a6e WatchSource:0}: Error finding container 61660af84f1e67c1bd631095162eeb88e7f04bdc621a6c3109cf2397ed568a6e: Status 404 returned error can't find the container with id 61660af84f1e67c1bd631095162eeb88e7f04bdc621a6c3109cf2397ed568a6e
Oct 30 22:36:53.267405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.267372    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"]
Oct 30 22:36:53.270631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.270592    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:53.270765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.270664    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:53.270765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.270694    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:36:53.518198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.518110    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:53.518704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.518684    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:53.519589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.519564    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:36:53.830055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.829942    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" event={"ID":"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f","Type":"ContainerStarted","Data":"80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d"}
Oct 30 22:36:53.830055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.830014    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" event={"ID":"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f","Type":"ContainerStarted","Data":"61660af84f1e67c1bd631095162eeb88e7f04bdc621a6c3109cf2397ed568a6e"}
Oct 30 22:36:53.872683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.872625    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" podStartSLOduration=1.8726077829999999 podStartE2EDuration="1.872607783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:53.871716542 +0000 UTC m=+1797.910655428" watchObservedRunningTime="2026-10-30 22:36:53.872607783 +0000 UTC m=+1797.911546670"
Oct 30 22:36:53.921809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:53.921774    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:54.928264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:54.928212    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:55.849815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:55.849780    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:55.850747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:55.850724    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:36:57.025197 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:57.025172    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:36:57.028723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:57.028701    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:36:58.430934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.430896    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"]
Oct 30 22:36:58.436067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.436043    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:58.449137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.449110    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bc48cc\""
Oct 30 22:36:58.458358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.458331    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bc48cc-dockercfg-rgj97\""
Oct 30 22:36:58.465297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.465267    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"]
Oct 30 22:36:58.597252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.596534    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nlf\" (UniqueName: \"kubernetes.io/projected/71ab1e84-d213-4926-acf5-1376f4cfc2fd-kube-api-access-q6nlf\") pod \"payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7\" (UID: \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:58.597252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.596689    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab1e84-d213-4926-acf5-1376f4cfc2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7\" (UID: \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:58.710292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.706604    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nlf\" (UniqueName: \"kubernetes.io/projected/71ab1e84-d213-4926-acf5-1376f4cfc2fd-kube-api-access-q6nlf\") pod \"payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7\" (UID: \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:58.710292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.706771    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab1e84-d213-4926-acf5-1376f4cfc2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7\" (UID: \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:58.710292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.707785    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab1e84-d213-4926-acf5-1376f4cfc2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7\" (UID: \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:58.836346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:58.835059    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nlf\" (UniqueName: \"kubernetes.io/projected/71ab1e84-d213-4926-acf5-1376f4cfc2fd-kube-api-access-q6nlf\") pod \"payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7\" (UID: \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:59.063302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:59.058522    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:59.458308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:59.457307    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"]
Oct 30 22:36:59.506294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:59.506266    4581 scope.go:122] "RemoveContainer" containerID="a5540e9df7235a724518302c7f20fafa629ff47502561b1bdeaf4fa5ed18010e"
Oct 30 22:36:59.952682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:59.952648    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7" event={"ID":"71ab1e84-d213-4926-acf5-1376f4cfc2fd","Type":"ContainerStarted","Data":"8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525"}
Oct 30 22:36:59.952883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:59.952702    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7" event={"ID":"71ab1e84-d213-4926-acf5-1376f4cfc2fd","Type":"ContainerStarted","Data":"fa54f7c90048d36cbe4a7a5ae049d27cd5a457e0fa65fc577d6eb13f1d8f49a8"}
Oct 30 22:36:59.953652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:59.953633    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:36:59.997349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:36:59.997294    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7" podStartSLOduration=1.997278443 podStartE2EDuration="1.997278443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:59.990620653 +0000 UTC m=+1804.029559539" watchObservedRunningTime="2026-10-30 22:36:59.997278443 +0000 UTC m=+1804.036217329"
Oct 30 22:37:00.556998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:00.556959    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"]
Oct 30 22:37:00.564517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:00.564480    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-62236a-cz646"]
Oct 30 22:37:02.103862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:02.103826    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e7d2dd-34ac-482d-a185-d5453c068c85" path="/var/lib/kubelet/pods/66e7d2dd-34ac-482d-a185-d5453c068c85/volumes"
Oct 30 22:37:04.570826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.570782    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"]
Oct 30 22:37:04.577238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.577195    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:04.593774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.593744    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-54c7f0\""
Oct 30 22:37:04.594051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.593617    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-54c7f0-dockercfg-pjkz9\""
Oct 30 22:37:04.601664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.601632    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"]
Oct 30 22:37:04.631976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.631932    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"]
Oct 30 22:37:04.638192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.638161    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:04.649857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.647908    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrtq\" (UniqueName: \"kubernetes.io/projected/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-kube-api-access-qrrtq\") pod \"payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m\" (UID: \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:04.649857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.648138    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m\" (UID: \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:04.663295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.662774    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"]
Oct 30 22:37:04.778249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.763380    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrtq\" (UniqueName: \"kubernetes.io/projected/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-kube-api-access-qrrtq\") pod \"payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m\" (UID: \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:04.778249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.763695    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psmsr\" (UniqueName: \"kubernetes.io/projected/710874c3-6407-406c-9955-5c3b9c2aa2c0-kube-api-access-psmsr\") pod \"payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn\" (UID: \"710874c3-6407-406c-9955-5c3b9c2aa2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:04.778249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.763739    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m\" (UID: \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:04.778249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.763865    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710874c3-6407-406c-9955-5c3b9c2aa2c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn\" (UID: \"710874c3-6407-406c-9955-5c3b9c2aa2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:04.796243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.792134    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m\" (UID: \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:04.917246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.909808    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmsr\" (UniqueName: \"kubernetes.io/projected/710874c3-6407-406c-9955-5c3b9c2aa2c0-kube-api-access-psmsr\") pod \"payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn\" (UID: \"710874c3-6407-406c-9955-5c3b9c2aa2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:04.917246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.909910    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710874c3-6407-406c-9955-5c3b9c2aa2c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn\" (UID: \"710874c3-6407-406c-9955-5c3b9c2aa2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:04.917246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:04.910914    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710874c3-6407-406c-9955-5c3b9c2aa2c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn\" (UID: \"710874c3-6407-406c-9955-5c3b9c2aa2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:05.041255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:05.037642    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmsr\" (UniqueName: \"kubernetes.io/projected/710874c3-6407-406c-9955-5c3b9c2aa2c0-kube-api-access-psmsr\") pod \"payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn\" (UID: \"710874c3-6407-406c-9955-5c3b9c2aa2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:05.064245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:05.052428    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrtq\" (UniqueName: \"kubernetes.io/projected/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-kube-api-access-qrrtq\") pod \"payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m\" (UID: \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:05.194905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:05.194792    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:05.275542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:05.275514    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:05.516316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:05.516278    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"]
Oct 30 22:37:05.516749 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:37:05.516722    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc54dee_efa4_49ef_84c2_d0d0c3c3ce66.slice/crio-ee2cf6fc94da8fcde64504ea50e868a6c4b4904802e1a7e5a5d3fc57dfe9a7a7 WatchSource:0}: Error finding container ee2cf6fc94da8fcde64504ea50e868a6c4b4904802e1a7e5a5d3fc57dfe9a7a7: Status 404 returned error can't find the container with id ee2cf6fc94da8fcde64504ea50e868a6c4b4904802e1a7e5a5d3fc57dfe9a7a7
Oct 30 22:37:05.621627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:05.621596    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"]
Oct 30 22:37:05.622708 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:37:05.622682    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710874c3_6407_406c_9955_5c3b9c2aa2c0.slice/crio-79bed5c52cdecec822c759aea00cfcc08456b1f45c2c2ada080653878ac3007c WatchSource:0}: Error finding container 79bed5c52cdecec822c759aea00cfcc08456b1f45c2c2ada080653878ac3007c: Status 404 returned error can't find the container with id 79bed5c52cdecec822c759aea00cfcc08456b1f45c2c2ada080653878ac3007c
Oct 30 22:37:06.077944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:06.077901    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m" event={"ID":"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66","Type":"ContainerStarted","Data":"762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5"}
Oct 30 22:37:06.078146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:06.077971    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:06.078146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:06.077995    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m" event={"ID":"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66","Type":"ContainerStarted","Data":"ee2cf6fc94da8fcde64504ea50e868a6c4b4904802e1a7e5a5d3fc57dfe9a7a7"}
Oct 30 22:37:06.079863 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:06.079835    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn" event={"ID":"710874c3-6407-406c-9955-5c3b9c2aa2c0","Type":"ContainerStarted","Data":"87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9"}
Oct 30 22:37:06.080006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:06.079878    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn" event={"ID":"710874c3-6407-406c-9955-5c3b9c2aa2c0","Type":"ContainerStarted","Data":"79bed5c52cdecec822c759aea00cfcc08456b1f45c2c2ada080653878ac3007c"}
Oct 30 22:37:06.080006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:06.079984    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:06.116602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:06.116544    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m" podStartSLOduration=2.116529224 podStartE2EDuration="2.116529224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:06.112557588 +0000 UTC m=+1810.151496474" watchObservedRunningTime="2026-10-30 22:37:06.116529224 +0000 UTC m=+1810.155468137"
Oct 30 22:37:06.156512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:06.156458    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn" podStartSLOduration=2.15644368 podStartE2EDuration="2.15644368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:06.148259918 +0000 UTC m=+1810.187198800" watchObservedRunningTime="2026-10-30 22:37:06.15644368 +0000 UTC m=+1810.195382566"
Oct 30 22:37:07.829458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:07.829417    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"]
Oct 30 22:37:07.829933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:07.829730    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" containerID="cri-o://69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8" gracePeriod=30
Oct 30 22:37:09.121241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.121187    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"]
Oct 30 22:37:09.128436 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.128404    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.150196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.150016    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"]
Oct 30 22:37:09.153057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.152742    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6e0edc-openshift-default-dockercfg-pkdjc\""
Oct 30 22:37:09.230798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.230752    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.230997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.230810    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sq9h\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-kube-api-access-2sq9h\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.230997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.230935    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.230997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.230968    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a61397-afcd-4557-80c6-de16c4023d99-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.230997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.230985    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.231290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.231212    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.231358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.231333    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a61397-afcd-4557-80c6-de16c4023d99-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.231429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.231396    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.231484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.231431    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340132    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340202    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq9h\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-kube-api-access-2sq9h\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340303    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340331    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a61397-afcd-4557-80c6-de16c4023d99-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340355    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340487    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340526    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a61397-afcd-4557-80c6-de16c4023d99-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340560    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.340593    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.341151    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.341983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.341439    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.342054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.342011    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.363243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.362151    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.385299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.383768    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a61397-afcd-4557-80c6-de16c4023d99-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.425291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.415542    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sq9h\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-kube-api-access-2sq9h\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.442475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.442431    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.459271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.445981    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.472249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.465841    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a61397-afcd-4557-80c6-de16c4023d99-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:09.750103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:09.750007    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:10.100321 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:37:10.100288    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a61397_afcd_4557_80c6_de16c4023d99.slice/crio-dd0b9fdc9872cc6c9a97a5b5695d36a7ae040e6a489ec3f54d195785cc1d90bc WatchSource:0}: Error finding container dd0b9fdc9872cc6c9a97a5b5695d36a7ae040e6a489ec3f54d195785cc1d90bc: Status 404 returned error can't find the container with id dd0b9fdc9872cc6c9a97a5b5695d36a7ae040e6a489ec3f54d195785cc1d90bc
Oct 30 22:37:10.103519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:10.103473    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:10.103657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:10.103568    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:10.103657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:10.103611    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:10.105507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:10.105481    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"]
Oct 30 22:37:10.218029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:10.217988    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" event={"ID":"f7a61397-afcd-4557-80c6-de16c4023d99","Type":"ContainerStarted","Data":"94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411"}
Oct 30 22:37:10.218500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:10.218045    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" event={"ID":"f7a61397-afcd-4557-80c6-de16c4023d99","Type":"ContainerStarted","Data":"dd0b9fdc9872cc6c9a97a5b5695d36a7ae040e6a489ec3f54d195785cc1d90bc"}
Oct 30 22:37:10.265446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:10.265364    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" podStartSLOduration=1.265345362 podStartE2EDuration="1.265345362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:10.260045338 +0000 UTC m=+1814.298984224" watchObservedRunningTime="2026-10-30 22:37:10.265345362 +0000 UTC m=+1814.304284249"
Oct 30 22:37:10.751244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:10.750326    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:10.821386 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:37:10.821342    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:37:11.759111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:11.759071    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:11.999900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:11.999864    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:37:12.236363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:12.236324    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:12.237363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:12.237344    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:37:13.155479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.155448    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:37:13.209259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209187    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-certs\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.209446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209308    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-envoy\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.209446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209404    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-podinfo\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.209583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209496    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-certs" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:13.209583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209543    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-token\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-token\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.209717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209698    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-kube-api-access-lxn5j\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-kube-api-access-lxn5j\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.209777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209744    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-socket\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.209818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209776    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istiod-ca-cert\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.209885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209867    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-credential-socket\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.209945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.209905    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-data\") pod \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\" (UID: \"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4\") "
Oct 30 22:37:13.212314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.210881    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:13.212314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.211890    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-credential-socket" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:13.212314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.211910    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-data" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:13.212570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.212316    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-socket" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:13.212964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.212939    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-envoy" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:13.213372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.213342    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istiod-ca-cert" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:13.213511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.213487    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-token" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:13.214393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.214362    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-podinfo" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:37:13.214508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.214408    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-kube-api-access-lxn5j" pod "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" (UID: "b519829a-bcbe-4c2e-a4ca-28b4bd299dd4"). InnerVolumeSpecName "kube-api-access-lxn5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:13.251475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.251439    4581 generic.go:356] "Generic (PLEG): container finished" podID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerID="69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8" exitCode=0
Oct 30 22:37:13.251691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.251523    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"
Oct 30 22:37:13.251691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.251534    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" event={"ID":"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4","Type":"ContainerDied","Data":"69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8"}
Oct 30 22:37:13.251691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.251631    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" event={"ID":"b519829a-bcbe-4c2e-a4ca-28b4bd299dd4","Type":"ContainerDied","Data":"a3a513099ee3b18c83af7d93258d77f948c13b3343d07d107f3456769845b737"}
Oct 30 22:37:13.251691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.251660    4581 scope.go:122] "RemoveContainer" containerID="69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8"
Oct 30 22:37:13.270043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.269977    4581 scope.go:122] "RemoveContainer" containerID="7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8"
Oct 30 22:37:13.286505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.286473    4581 scope.go:122] "RemoveContainer" containerID="69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8"
Oct 30 22:37:13.286898 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:37:13.286876    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8\": container with ID starting with 69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8 not found: ID does not exist" containerID="69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8"
Oct 30 22:37:13.286981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.286909    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8"} err="failed to get container status \"69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8\": rpc error: code = NotFound desc = could not find container \"69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8\": container with ID starting with 69435caa0e698c8fa03bd4a8a79e52c7e09bb1625b36f8c639f130c2150581e8 not found: ID does not exist"
Oct 30 22:37:13.286981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.286931    4581 scope.go:122] "RemoveContainer" containerID="7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8"
Oct 30 22:37:13.287296 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:37:13.287265    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8\": container with ID starting with 7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8 not found: ID does not exist" containerID="7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8"
Oct 30 22:37:13.287351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.287308    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8"} err="failed to get container status \"7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8\": rpc error: code = NotFound desc = could not find container \"7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8\": container with ID starting with 7407fed6068589b2f76f34c2021d54c02a0c622c3fd8863d0d77cd120187f0a8 not found: ID does not exist"
Oct 30 22:37:13.306271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.306200    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"]
Oct 30 22:37:13.311259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.311200    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl"]
Oct 30 22:37:13.312278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.312249    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxn5j\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-kube-api-access-lxn5j\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:13.312278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.312281    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:13.312465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.312296    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:13.312465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.312310    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:13.312465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.312322    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:13.312465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.312331    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:13.312465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.312339    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:13.312465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:13.312348    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:37:14.101542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.101508    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" path="/var/lib/kubelet/pods/b519829a-bcbe-4c2e-a4ca-28b4bd299dd4/volumes"
Oct 30 22:37:14.105181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.105143    4581 patch_prober.go:29] interesting pod/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:37:14.105332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.105205    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-62236a-openshift-default-899986fcf-8kdtl" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:37:14.732922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.732882    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"]
Oct 30 22:37:14.733996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.733969    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy"
Oct 30 22:37:14.734647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.733999    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy"
Oct 30 22:37:14.734647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.734232    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy"
Oct 30 22:37:14.734647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.734256    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy"
Oct 30 22:37:14.739962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.739934    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:14.764926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.764886    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6e0edc\""
Oct 30 22:37:14.766438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.765641    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6e0edc-dockercfg-ddrt9\""
Oct 30 22:37:14.766438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.765980    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"]
Oct 30 22:37:14.856256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.842386    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9cc6f1-9ec2-420f-b112-405e97693165-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s\" (UID: \"de9cc6f1-9ec2-420f-b112-405e97693165\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:14.856256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.842514    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9965l\" (UniqueName: \"kubernetes.io/projected/de9cc6f1-9ec2-420f-b112-405e97693165-kube-api-access-9965l\") pod \"payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s\" (UID: \"de9cc6f1-9ec2-420f-b112-405e97693165\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:14.926331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.926286    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"]
Oct 30 22:37:14.939130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.931565    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy"
Oct 30 22:37:14.989310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.961307    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b519829a-bcbe-4c2e-a4ca-28b4bd299dd4" containerName="istio-proxy"
Oct 30 22:37:14.989570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.949424    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9cc6f1-9ec2-420f-b112-405e97693165-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s\" (UID: \"de9cc6f1-9ec2-420f-b112-405e97693165\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:14.995485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.995442    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9965l\" (UniqueName: \"kubernetes.io/projected/de9cc6f1-9ec2-420f-b112-405e97693165-kube-api-access-9965l\") pod \"payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s\" (UID: \"de9cc6f1-9ec2-420f-b112-405e97693165\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:15.036249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:14.950320    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9cc6f1-9ec2-420f-b112-405e97693165-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s\" (UID: \"de9cc6f1-9ec2-420f-b112-405e97693165\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:15.172292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.129648    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:15.216000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.213524    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9965l\" (UniqueName: \"kubernetes.io/projected/de9cc6f1-9ec2-420f-b112-405e97693165-kube-api-access-9965l\") pod \"payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s\" (UID: \"de9cc6f1-9ec2-420f-b112-405e97693165\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:15.229906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.229869    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"]
Oct 30 22:37:15.259754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.259645    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac035c3-3d22-48ff-bb75-4259779288d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8\" (UID: \"5ac035c3-3d22-48ff-bb75-4259779288d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:15.259982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.259951    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd77n\" (UniqueName: \"kubernetes.io/projected/5ac035c3-3d22-48ff-bb75-4259779288d6-kube-api-access-gd77n\") pod \"payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8\" (UID: \"5ac035c3-3d22-48ff-bb75-4259779288d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:15.362350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.362310    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac035c3-3d22-48ff-bb75-4259779288d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8\" (UID: \"5ac035c3-3d22-48ff-bb75-4259779288d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:15.362583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.362464    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd77n\" (UniqueName: \"kubernetes.io/projected/5ac035c3-3d22-48ff-bb75-4259779288d6-kube-api-access-gd77n\") pod \"payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8\" (UID: \"5ac035c3-3d22-48ff-bb75-4259779288d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:15.363195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.363174    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac035c3-3d22-48ff-bb75-4259779288d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8\" (UID: \"5ac035c3-3d22-48ff-bb75-4259779288d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:15.368598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.368552    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:15.383977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.383939    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd77n\" (UniqueName: \"kubernetes.io/projected/5ac035c3-3d22-48ff-bb75-4259779288d6-kube-api-access-gd77n\") pod \"payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8\" (UID: \"5ac035c3-3d22-48ff-bb75-4259779288d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:15.533023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.532936    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:15.719470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.719440    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"]
Oct 30 22:37:15.722544 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:37:15.722510    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9cc6f1_9ec2_420f_b112_405e97693165.slice/crio-e01668a050f79603d95a7d7ede4e3990feaa9552349a95c55639c401722bb0f8 WatchSource:0}: Error finding container e01668a050f79603d95a7d7ede4e3990feaa9552349a95c55639c401722bb0f8: Status 404 returned error can't find the container with id e01668a050f79603d95a7d7ede4e3990feaa9552349a95c55639c401722bb0f8
Oct 30 22:37:15.861680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:15.861639    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"]
Oct 30 22:37:15.863834 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:37:15.863798    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac035c3_3d22_48ff_bb75_4259779288d6.slice/crio-185e8d1171e3729ffe58fca566829efe362c2ca637461c37ae11bd9b2bfe4681 WatchSource:0}: Error finding container 185e8d1171e3729ffe58fca566829efe362c2ca637461c37ae11bd9b2bfe4681: Status 404 returned error can't find the container with id 185e8d1171e3729ffe58fca566829efe362c2ca637461c37ae11bd9b2bfe4681
Oct 30 22:37:16.291397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:16.291358    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8" event={"ID":"5ac035c3-3d22-48ff-bb75-4259779288d6","Type":"ContainerStarted","Data":"862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34"}
Oct 30 22:37:16.291613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:16.291424    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8" event={"ID":"5ac035c3-3d22-48ff-bb75-4259779288d6","Type":"ContainerStarted","Data":"185e8d1171e3729ffe58fca566829efe362c2ca637461c37ae11bd9b2bfe4681"}
Oct 30 22:37:16.291613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:16.291507    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:16.293586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:16.293559    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s" event={"ID":"de9cc6f1-9ec2-420f-b112-405e97693165","Type":"ContainerStarted","Data":"828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e"}
Oct 30 22:37:16.293706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:16.293602    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s" event={"ID":"de9cc6f1-9ec2-420f-b112-405e97693165","Type":"ContainerStarted","Data":"e01668a050f79603d95a7d7ede4e3990feaa9552349a95c55639c401722bb0f8"}
Oct 30 22:37:16.293706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:16.293688    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:16.345369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:16.345261    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8" podStartSLOduration=2.345242727 podStartE2EDuration="2.345242727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:16.340527123 +0000 UTC m=+1820.379466008" watchObservedRunningTime="2026-10-30 22:37:16.345242727 +0000 UTC m=+1820.384181615"
Oct 30 22:37:16.386267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:16.386188    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s" podStartSLOduration=2.386172486 podStartE2EDuration="2.386172486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:16.380621336 +0000 UTC m=+1820.419560226" watchObservedRunningTime="2026-10-30 22:37:16.386172486 +0000 UTC m=+1820.425111372"
Oct 30 22:37:17.096462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:17.096429    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:37:17.096987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:17.096563    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:37:23.109295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.108504    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"]
Oct 30 22:37:23.124081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.114736    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.194627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.194582    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f0bb45-openshift-default-dockercfg-f5d62\""
Oct 30 22:37:23.264245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.256486    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"]
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.602273    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-podinfo\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.605774    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-data\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.606652    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.607313    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-envoy\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.607385    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-credential-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.607428    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvng\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-kube-api-access-jdvng\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.607474    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc66240-9c5e-43f8-9db0-560736cbe443-istiod-ca-cert\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.607533    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-token\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.622633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.607563    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-certs\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709488 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709450    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709602    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-envoy\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709637    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-credential-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709660    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvng\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-kube-api-access-jdvng\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709685    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc66240-9c5e-43f8-9db0-560736cbe443-istiod-ca-cert\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709719    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-token\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709737    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-certs\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709792    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-podinfo\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709811    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-data\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.709934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.709908    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.710195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.710128    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-data\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.710559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.710535    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc66240-9c5e-43f8-9db0-560736cbe443-istiod-ca-cert\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.710616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.710563    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-certs\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.711045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.711018    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-credential-socket\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.712953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.712929    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-envoy\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.713349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.713331    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-podinfo\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.729250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.729194    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-token\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.736343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.736306    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvng\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-kube-api-access-jdvng\") pod \"e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") " pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:23.819270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:23.814897    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:24.223862 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:37:24.223828    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc66240_9c5e_43f8_9db0_560736cbe443.slice/crio-e83ad8432fef9f00f1e16839b6c52d156f50258bbb51a39d72ba9185f43b2419 WatchSource:0}: Error finding container e83ad8432fef9f00f1e16839b6c52d156f50258bbb51a39d72ba9185f43b2419: Status 404 returned error can't find the container with id e83ad8432fef9f00f1e16839b6c52d156f50258bbb51a39d72ba9185f43b2419
Oct 30 22:37:24.224194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:24.223991    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"]
Oct 30 22:37:24.227293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:24.227257    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:24.227400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:24.227327    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:24.227400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:24.227357    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:24.643470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:24.643420    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" event={"ID":"cbc66240-9c5e-43f8-9db0-560736cbe443","Type":"ContainerStarted","Data":"97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6"}
Oct 30 22:37:24.643805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:24.643781    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" event={"ID":"cbc66240-9c5e-43f8-9db0-560736cbe443","Type":"ContainerStarted","Data":"e83ad8432fef9f00f1e16839b6c52d156f50258bbb51a39d72ba9185f43b2419"}
Oct 30 22:37:24.794247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:24.791627    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podStartSLOduration=1.791604135 podStartE2EDuration="1.791604135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:24.780140039 +0000 UTC m=+1828.819078927" watchObservedRunningTime="2026-10-30 22:37:24.791604135 +0000 UTC m=+1828.830543022"
Oct 30 22:37:24.817177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:24.817145    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:25.821778 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:25.821750    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:26.697708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:26.697675    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:26.699073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:26.699044    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:37:27.315483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:27.315453    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:37:27.315958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:27.315528    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:37:31.943024 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:31.942991    4581 generic.go:356] "Generic (PLEG): container finished" podID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerID="fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0" exitCode=0
Oct 30 22:37:31.943569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:31.943060    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" event={"ID":"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56","Type":"ContainerDied","Data":"fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0"}
Oct 30 22:37:31.943645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:31.943601    4581 scope.go:122] "RemoveContainer" containerID="fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0"
Oct 30 22:37:31.945750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:31.945718    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:31.945829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:31.945791    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:31.945829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:31.945825    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:32.989133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:32.989085    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" event={"ID":"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56","Type":"ContainerStarted","Data":"b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59"}
Oct 30 22:37:33.131547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:33.131516    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:37:33.131929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:33.131907    4581 patch_prober.go:29] interesting pod/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:37:33.132069 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:33.132047    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 22:37:33.603762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:33.603731    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:37:34.120144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:34.120106    4581 patch_prober.go:29] interesting pod/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:37:34.120602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:34.120161    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 22:37:35.120468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:35.120435    4581 patch_prober.go:29] interesting pod/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:37:35.120900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:35.120489    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 22:37:36.120149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:36.120058    4581 patch_prober.go:29] interesting pod/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:37:36.120361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:36.120166    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 22:37:37.124242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:37.120001    4581 patch_prober.go:29] interesting pod/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:37:37.124242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:37.120064    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 22:37:38.120867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:38.120822    4581 patch_prober.go:29] interesting pod/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:37:38.121082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:38.120904    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 22:37:39.126588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:39.126558    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:37:39.127714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:39.127695    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:37:40.820774 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:37:40.820731    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:37:55.069579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.069545    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"]
Oct 30 22:37:55.240047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.240013    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.253283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.240619    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"]
Oct 30 22:37:55.275517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.275476    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c13db0-openshift-default-dockercfg-ppd9k\""
Oct 30 22:37:55.337744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.337649    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-credential-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.337744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.337703    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-certs\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.337744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.337724    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-token\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.338052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.337748    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-podinfo\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.338052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.337966    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istiod-ca-cert\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.338052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.338036    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-data\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.338265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.338206    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbx2\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-kube-api-access-qvbx2\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.338332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.338310    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.338545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.338528    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-envoy\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.440697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.440661    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-data\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.440893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.440737    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbx2\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-kube-api-access-qvbx2\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.440893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.440767    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.440988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.440908    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-envoy\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441075    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-credential-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441117    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-certs\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441073    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-data\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441144    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-token\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441164    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441169    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-podinfo\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441315    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-credential-socket\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441361    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istiod-ca-cert\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.441521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.441471    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-certs\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.442326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.442288    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istiod-ca-cert\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.443523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.443503    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-envoy\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.443919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.443899    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-podinfo\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.453343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.453313    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"]
Oct 30 22:37:55.453607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.453560    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy" containerID="cri-o://b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8" gracePeriod=30
Oct 30 22:37:55.464553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.464138    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbx2\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-kube-api-access-qvbx2\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.465615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.465587    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-token\") pod \"e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") " pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:55.592748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:55.592641    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:56.003747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:56.003716    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"]
Oct 30 22:37:56.007774 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:37:56.007741    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9bdb1e_eaf9_410d_9fe5_3337138ca4db.slice/crio-4b81eb896bb8dd3b3b3a45c993b3bf169639ad723a985e4c91fb9b211c79c739 WatchSource:0}: Error finding container 4b81eb896bb8dd3b3b3a45c993b3bf169639ad723a985e4c91fb9b211c79c739: Status 404 returned error can't find the container with id 4b81eb896bb8dd3b3b3a45c993b3bf169639ad723a985e4c91fb9b211c79c739
Oct 30 22:37:56.011596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:56.011549    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:56.011689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:56.011650    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:56.011729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:56.011700    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:37:56.296207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:56.296100    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" event={"ID":"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db","Type":"ContainerStarted","Data":"c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d"}
Oct 30 22:37:56.296207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:56.296173    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" event={"ID":"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db","Type":"ContainerStarted","Data":"4b81eb896bb8dd3b3b3a45c993b3bf169639ad723a985e4c91fb9b211c79c739"}
Oct 30 22:37:56.338980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:56.338909    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podStartSLOduration=2.338889528 podStartE2EDuration="2.338889528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:56.337127482 +0000 UTC m=+1860.376066369" watchObservedRunningTime="2026-10-30 22:37:56.338889528 +0000 UTC m=+1860.377828415"
Oct 30 22:37:56.597532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:56.597416    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:57.607251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:57.606507    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:58.318950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:58.318920    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:37:58.319896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:37:58.319872    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:38:00.677486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.676417    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"]
Oct 30 22:38:00.685081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.684103    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:00.741386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.730086    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c13db0\""
Oct 30 22:38:00.755313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.730471    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c13db0-dockercfg-nzlt9\""
Oct 30 22:38:00.771246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.758359    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a025092a-ad46-4269-950a-21b5b7a0447d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn\" (UID: \"a025092a-ad46-4269-950a-21b5b7a0447d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:00.771246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.758558    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zkb\" (UniqueName: \"kubernetes.io/projected/a025092a-ad46-4269-950a-21b5b7a0447d-kube-api-access-t9zkb\") pod \"payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn\" (UID: \"a025092a-ad46-4269-950a-21b5b7a0447d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:00.925254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.892414    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a025092a-ad46-4269-950a-21b5b7a0447d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn\" (UID: \"a025092a-ad46-4269-950a-21b5b7a0447d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:00.925254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.892620    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zkb\" (UniqueName: \"kubernetes.io/projected/a025092a-ad46-4269-950a-21b5b7a0447d-kube-api-access-t9zkb\") pod \"payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn\" (UID: \"a025092a-ad46-4269-950a-21b5b7a0447d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:00.925254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.893923    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a025092a-ad46-4269-950a-21b5b7a0447d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn\" (UID: \"a025092a-ad46-4269-950a-21b5b7a0447d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:00.925254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:00.893978    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"]
Oct 30 22:38:01.141540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.141482    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zkb\" (UniqueName: \"kubernetes.io/projected/a025092a-ad46-4269-950a-21b5b7a0447d-kube-api-access-t9zkb\") pod \"payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn\" (UID: \"a025092a-ad46-4269-950a-21b5b7a0447d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:01.204321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.204284    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:38:01.308848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.308811    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istiod-ca-cert\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.308925    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-podinfo\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.308977    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-credential-socket\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309012    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-certs\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309038    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-socket\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309074    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-token\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-token\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309207    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-kube-api-access-ldp9q\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-kube-api-access-ldp9q\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309209    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istiod-ca-cert" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:01.309349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309296    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-envoy\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309339    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-data\") pod \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\" (UID: \"ee0e8a04-1c10-4718-9e50-7816c5d5c35c\") "
Oct 30 22:38:01.309570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309373    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-credential-socket" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.309570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309437    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-socket" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.309570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309453    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-certs" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.309761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.309737    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-data" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.310318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.310294    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.310438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.310319    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.310438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.310335    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.310438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.310350    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.310438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.310363    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.311921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.311891    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-token" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:01.311921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.311911    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-envoy" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.312110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.312093    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-kube-api-access-ldp9q" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "kube-api-access-ldp9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:01.312178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.312126    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-podinfo" pod "ee0e8a04-1c10-4718-9e50-7816c5d5c35c" (UID: "ee0e8a04-1c10-4718-9e50-7816c5d5c35c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:38:01.347794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.347761    4581 generic.go:356] "Generic (PLEG): container finished" podID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerID="b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8" exitCode=0
Oct 30 22:38:01.347967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.347812    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" event={"ID":"ee0e8a04-1c10-4718-9e50-7816c5d5c35c","Type":"ContainerDied","Data":"b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8"}
Oct 30 22:38:01.347967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.347868    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" event={"ID":"ee0e8a04-1c10-4718-9e50-7816c5d5c35c","Type":"ContainerDied","Data":"f63740eaa4fda2e0def0e034e0fcd9e34a8d18c8c2a7d8fe6469a5c262a20ee3"}
Oct 30 22:38:01.347967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.347817    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"
Oct 30 22:38:01.347967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.347897    4581 scope.go:122] "RemoveContainer" containerID="b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8"
Oct 30 22:38:01.376441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.376413    4581 scope.go:122] "RemoveContainer" containerID="b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8"
Oct 30 22:38:01.376823 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:38:01.376795    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8\": container with ID starting with b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8 not found: ID does not exist" containerID="b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8"
Oct 30 22:38:01.376938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.376831    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8"} err="failed to get container status \"b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8\": rpc error: code = NotFound desc = could not find container \"b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8\": container with ID starting with b824e54f1379b05f02c1fb43f5474aefd67c1db9d76822af8507c35002e693a8 not found: ID does not exist"
Oct 30 22:38:01.407022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.406943    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"]
Oct 30 22:38:01.412093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.412064    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.412093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.412096    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.412327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.412111    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldp9q\" (UniqueName: \"kubernetes.io/projected/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-kube-api-access-ldp9q\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.412327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.412120    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0e8a04-1c10-4718-9e50-7816c5d5c35c-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.414203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.414179    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4"]
Oct 30 22:38:01.423595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.423562    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:01.789649 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.789622    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"]
Oct 30 22:38:01.790465 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:38:01.790432    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda025092a_ad46_4269_950a_21b5b7a0447d.slice/crio-f4d26e388dfa3e84c63489026bfa7a6e368017192e008c36c5baf3d9c65a92f4 WatchSource:0}: Error finding container f4d26e388dfa3e84c63489026bfa7a6e368017192e008c36c5baf3d9c65a92f4: Status 404 returned error can't find the container with id f4d26e388dfa3e84c63489026bfa7a6e368017192e008c36c5baf3d9c65a92f4
Oct 30 22:38:01.855683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.855659    4581 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:38:01.855832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:01.855707    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-582w4" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:38:02.112674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:02.112632    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" path="/var/lib/kubelet/pods/ee0e8a04-1c10-4718-9e50-7816c5d5c35c/volumes"
Oct 30 22:38:02.363927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:02.363830    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn" event={"ID":"a025092a-ad46-4269-950a-21b5b7a0447d","Type":"ContainerStarted","Data":"56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5"}
Oct 30 22:38:02.363927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:02.363897    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn" event={"ID":"a025092a-ad46-4269-950a-21b5b7a0447d","Type":"ContainerStarted","Data":"f4d26e388dfa3e84c63489026bfa7a6e368017192e008c36c5baf3d9c65a92f4"}
Oct 30 22:38:02.364188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:02.363941    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:02.402682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:02.402619    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn" podStartSLOduration=2.40260146 podStartE2EDuration="2.40260146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:02.397258648 +0000 UTC m=+1866.436197539" watchObservedRunningTime="2026-10-30 22:38:02.40260146 +0000 UTC m=+1866.441540373"
Oct 30 22:38:10.843630 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:38:10.843594    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:38:13.379419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:13.379384    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:38:16.181452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:16.181404    4581 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:38:16.182006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:16.181492    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:38:18.771539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.771503    4581 generic.go:356] "Generic (PLEG): container finished" podID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerID="80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d" exitCode=0
Oct 30 22:38:18.772106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.771581    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" event={"ID":"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f","Type":"ContainerDied","Data":"80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d"}
Oct 30 22:38:18.772194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.772100    4581 scope.go:122] "RemoveContainer" containerID="80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d"
Oct 30 22:38:18.773920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.773891    4581 generic.go:356] "Generic (PLEG): container finished" podID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerID="6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce" exitCode=0
Oct 30 22:38:18.774154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.774137    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" event={"ID":"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d","Type":"ContainerDied","Data":"6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce"}
Oct 30 22:38:18.774646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.774625    4581 scope.go:122] "RemoveContainer" containerID="6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce"
Oct 30 22:38:18.774742 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.774669    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:18.774812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.774753    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:18.774812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.774804    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:18.776473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.776445    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:18.776587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.776509    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:18.776587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:18.776545    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:19.806577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:19.806539    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" event={"ID":"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f","Type":"ContainerStarted","Data":"3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0"}
Oct 30 22:38:19.817707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:19.817663    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" event={"ID":"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d","Type":"ContainerStarted","Data":"00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d"}
Oct 30 22:38:19.922163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:19.922123    4581 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:38:19.922370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:19.922188    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:38:19.922491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:19.922472    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:38:20.723737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:20.723706    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:38:20.723955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:20.723851    4581 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:38:20.723955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:20.723896    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:38:20.921608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:20.921549    4581 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:38:20.922104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:20.921628    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:38:21.723330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:21.723295    4581 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:38:21.723555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:21.723353    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:38:21.924246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:21.922083    4581 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:38:21.924246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:21.922145    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:38:22.723826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:22.723788    4581 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:38:22.724051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:22.723846    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:38:22.922076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:22.922040    4581 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:38:22.922281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:22.922105    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:38:23.724043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:23.724006    4581 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:38:23.724715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:23.724684    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:38:23.921530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:23.921403    4581 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:38:23.921530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:23.921466    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:38:24.723278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:24.723206    4581 patch_prober.go:29] interesting pod/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:38:24.723536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:24.723510    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:38:24.931619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:24.931571    4581 patch_prober.go:29] interesting pod/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:38:24.932289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:24.932211    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 22:38:25.729235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:25.729183    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:38:25.729729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:25.729702    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:38:25.730569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:25.730549    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:38:25.850376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:25.850340    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:38:25.927924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:25.927891    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:38:25.960474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:25.960436    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:38:31.063673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:31.063639    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"]
Oct 30 22:38:31.107569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:31.107530    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ef682c-p767k"]
Oct 30 22:38:32.102076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:32.102043    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a720020-91b0-4d87-97f5-c8418dd80cd3" path="/var/lib/kubelet/pods/7a720020-91b0-4d87-97f5-c8418dd80cd3/volumes"
Oct 30 22:38:40.147874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.147837    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"]
Oct 30 22:38:40.149035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.149013    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy"
Oct 30 22:38:40.149119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.149039    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy"
Oct 30 22:38:40.149285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.149264    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0e8a04-1c10-4718-9e50-7816c5d5c35c" containerName="istio-proxy"
Oct 30 22:38:40.187662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.187628    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"]
Oct 30 22:38:40.187902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.187740    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.202427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.202401    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-867f12-openshift-default-dockercfg-tfg6j\""
Oct 30 22:38:40.317269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.249852    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.320244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.320179    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.320655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.320596    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.320962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.320920    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlcs\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-kube-api-access-mzlcs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.321207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.321154    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.321555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.321534    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.321931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.321909    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a451d097-a43a-42eb-beaa-e7b394e8efb9-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.323707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.323687    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.324976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.324508    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.431185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.431076    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.431492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.431469    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.431662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.431646    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlcs\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-kube-api-access-mzlcs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.431793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.431779    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.431942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.431928    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.432133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.432118    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a451d097-a43a-42eb-beaa-e7b394e8efb9-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.432275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.432258    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.432513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.432498    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.432987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.432953    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.433099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.432990    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.433260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.432969    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.433260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.433249    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.433452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.433274    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.433452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.433383    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a451d097-a43a-42eb-beaa-e7b394e8efb9-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.435519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.435499    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.436039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.436020    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.463313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.453709    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.463313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.453722    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlcs\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-kube-api-access-mzlcs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.516287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.514747    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:40.862735 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:38:40.862694    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:38:40.938671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.938636    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"]
Oct 30 22:38:40.939462 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:38:40.939431    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda451d097_a43a_42eb_beaa_e7b394e8efb9.slice/crio-c23658f0644e5e1a8adec4a596f2c276c21df3b45a0c32637d2cff87d0928a78 WatchSource:0}: Error finding container c23658f0644e5e1a8adec4a596f2c276c21df3b45a0c32637d2cff87d0928a78: Status 404 returned error can't find the container with id c23658f0644e5e1a8adec4a596f2c276c21df3b45a0c32637d2cff87d0928a78
Oct 30 22:38:40.942777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.942728    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:40.942907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.942828    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:40.942907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:40.942872    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:38:41.133933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:41.133834    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" event={"ID":"a451d097-a43a-42eb-beaa-e7b394e8efb9","Type":"ContainerStarted","Data":"798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c"}
Oct 30 22:38:41.133933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:41.133901    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" event={"ID":"a451d097-a43a-42eb-beaa-e7b394e8efb9","Type":"ContainerStarted","Data":"c23658f0644e5e1a8adec4a596f2c276c21df3b45a0c32637d2cff87d0928a78"}
Oct 30 22:38:41.516031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:41.515944    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:42.522855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:42.522827    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:42.581236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:42.581162    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podStartSLOduration=2.581141775 podStartE2EDuration="2.581141775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:41.18751554 +0000 UTC m=+1905.226454428" watchObservedRunningTime="2026-10-30 22:38:42.581141775 +0000 UTC m=+1906.620080661"
Oct 30 22:38:43.206915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:43.206887    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:43.208032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:43.208011    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:38:45.577136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.577097    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"]
Oct 30 22:38:45.582369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.582345    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:45.606587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.606547    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"]
Oct 30 22:38:45.608170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.608075    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-867f12-dockercfg-zlfjb\""
Oct 30 22:38:45.623359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.623320    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-867f12\""
Oct 30 22:38:45.807244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.800322    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca001362-beab-4a6e-858b-bd735928f157-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q\" (UID: \"ca001362-beab-4a6e-858b-bd735928f157\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:45.807244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.800402    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhj7v\" (UniqueName: \"kubernetes.io/projected/ca001362-beab-4a6e-858b-bd735928f157-kube-api-access-vhj7v\") pod \"payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q\" (UID: \"ca001362-beab-4a6e-858b-bd735928f157\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:45.912456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.912410    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca001362-beab-4a6e-858b-bd735928f157-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q\" (UID: \"ca001362-beab-4a6e-858b-bd735928f157\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:45.912813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.912792    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhj7v\" (UniqueName: \"kubernetes.io/projected/ca001362-beab-4a6e-858b-bd735928f157-kube-api-access-vhj7v\") pod \"payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q\" (UID: \"ca001362-beab-4a6e-858b-bd735928f157\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:45.915320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:45.915294    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca001362-beab-4a6e-858b-bd735928f157-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q\" (UID: \"ca001362-beab-4a6e-858b-bd735928f157\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:46.004296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:46.000383    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhj7v\" (UniqueName: \"kubernetes.io/projected/ca001362-beab-4a6e-858b-bd735928f157-kube-api-access-vhj7v\") pod \"payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q\" (UID: \"ca001362-beab-4a6e-858b-bd735928f157\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:46.230292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:46.229664    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:46.644922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:46.644881    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"]
Oct 30 22:38:46.647148 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:38:46.647118    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca001362_beab_4a6e_858b_bd735928f157.slice/crio-1e5fbaed011eb3d927cd51bc7a2f31b33daa1c8d06eda17401d14d3fd1030821 WatchSource:0}: Error finding container 1e5fbaed011eb3d927cd51bc7a2f31b33daa1c8d06eda17401d14d3fd1030821: Status 404 returned error can't find the container with id 1e5fbaed011eb3d927cd51bc7a2f31b33daa1c8d06eda17401d14d3fd1030821
Oct 30 22:38:47.316195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:47.316162    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q" event={"ID":"ca001362-beab-4a6e-858b-bd735928f157","Type":"ContainerStarted","Data":"868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4"}
Oct 30 22:38:47.316426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:47.316231    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q" event={"ID":"ca001362-beab-4a6e-858b-bd735928f157","Type":"ContainerStarted","Data":"1e5fbaed011eb3d927cd51bc7a2f31b33daa1c8d06eda17401d14d3fd1030821"}
Oct 30 22:38:47.316426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:47.316263    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:47.376945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:47.365303    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q" podStartSLOduration=2.365282534 podStartE2EDuration="2.365282534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:47.351703789 +0000 UTC m=+1911.390642692" watchObservedRunningTime="2026-10-30 22:38:47.365282534 +0000 UTC m=+1911.404221420"
Oct 30 22:38:58.333201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:58.333167    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:38:59.610287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:59.610249    4581 scope.go:122] "RemoveContainer" containerID="3a2bc800b56baad9dd41113ab30cac48244094b1871d2c8e221e2e394dd2f05e"
Oct 30 22:38:59.623810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:38:59.623787    4581 scope.go:122] "RemoveContainer" containerID="b0555a09fbb00682bd65e41d19e8cf82f943c1a710538801a2c475fd4e7b8140"
Oct 30 22:39:01.328465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.328385    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"]
Oct 30 22:39:01.345205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.344964    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"]
Oct 30 22:39:01.345205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.345125    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.428989 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.428865    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3acfdb-openshift-default-dockercfg-bthnn\""
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455022    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-data\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455086    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-token\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455142    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6976r\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-kube-api-access-6976r\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455205    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-credential-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455256    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-envoy\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455376    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istiod-ca-cert\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455417    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455537    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-podinfo\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.472727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.455703    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-certs\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.697253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.671533    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-credential-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.776666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.776626    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-envoy\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.780988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.780899    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istiod-ca-cert\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.804252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.785336    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.804822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.804787    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-podinfo\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.805190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.805170    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-certs\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.805443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.805423    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-data\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.825358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.825321    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-token\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.825658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.825636    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6976r\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-kube-api-access-6976r\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.825936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.824274    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-certs\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.826059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.795055    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.826160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.824529    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-data\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.826281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.702272    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-credential-socket\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.833247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.782390    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istiod-ca-cert\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.893844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.889571    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6976r\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-kube-api-access-6976r\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.908638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.908543    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-envoy\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.910007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.909945    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-podinfo\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:01.937257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:01.934264    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-token\") pod \"e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") " pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:02.025766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:02.025663    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:02.623358 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:39:02.622903    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df7ae0f_f6b7_4a5d_bfd8_1faec99eebf2.slice/crio-740ef2eb6fe4f5e3ac0907317d49a69307ecf09b6d3ef846361ea937e96b5279 WatchSource:0}: Error finding container 740ef2eb6fe4f5e3ac0907317d49a69307ecf09b6d3ef846361ea937e96b5279: Status 404 returned error can't find the container with id 740ef2eb6fe4f5e3ac0907317d49a69307ecf09b6d3ef846361ea937e96b5279
Oct 30 22:39:02.624438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:02.624413    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"]
Oct 30 22:39:02.626543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:02.626509    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:02.626628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:02.626586    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:02.626628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:02.626614    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:02.891686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:02.891093    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" event={"ID":"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2","Type":"ContainerStarted","Data":"7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f"}
Oct 30 22:39:02.891686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:02.891164    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" event={"ID":"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2","Type":"ContainerStarted","Data":"740ef2eb6fe4f5e3ac0907317d49a69307ecf09b6d3ef846361ea937e96b5279"}
Oct 30 22:39:02.939925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:02.938563    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podStartSLOduration=1.938543071 podStartE2EDuration="1.938543071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:02.931872554 +0000 UTC m=+1926.970811452" watchObservedRunningTime="2026-10-30 22:39:02.938543071 +0000 UTC m=+1926.977481960"
Oct 30 22:39:03.028452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:03.028415    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:03.028633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:03.028488    4581 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 22:39:03.028633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:03.028561    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 22:39:04.034037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:04.034003    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:04.915297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:04.915210    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:04.916769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:04.916740    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:39:08.961124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:08.961089    4581 generic.go:356] "Generic (PLEG): container finished" podID="f7a61397-afcd-4557-80c6-de16c4023d99" containerID="94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411" exitCode=0
Oct 30 22:39:08.974290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:08.961259    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" event={"ID":"f7a61397-afcd-4557-80c6-de16c4023d99","Type":"ContainerDied","Data":"94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411"}
Oct 30 22:39:08.974290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:08.961926    4581 scope.go:122] "RemoveContainer" containerID="94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411"
Oct 30 22:39:08.974290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:08.964648    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:08.974290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:08.964728    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:08.974290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:08.964761    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:09.983243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:09.982303    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" event={"ID":"f7a61397-afcd-4557-80c6-de16c4023d99","Type":"ContainerStarted","Data":"e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25"}
Oct 30 22:39:09.993060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:09.988158    4581 generic.go:356] "Generic (PLEG): container finished" podID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerID="97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6" exitCode=0
Oct 30 22:39:09.993060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:09.988275    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" event={"ID":"cbc66240-9c5e-43f8-9db0-560736cbe443","Type":"ContainerDied","Data":"97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6"}
Oct 30 22:39:09.993060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:09.988789    4581 scope.go:122] "RemoveContainer" containerID="97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6"
Oct 30 22:39:09.993060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:09.991977    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:09.993060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:09.992080    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:09.993060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:09.992131    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:10.756262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:10.753408    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:39:10.756262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:10.753546    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:39:10.756262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:10.753591    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 22:39:10.894525 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:39:10.894458    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:39:11.003808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:11.003773    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" event={"ID":"cbc66240-9c5e-43f8-9db0-560736cbe443","Type":"ContainerStarted","Data":"ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7"}
Oct 30 22:39:11.698273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:11.698235    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:39:11.758654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:11.758616    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:39:11.759041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:11.759009    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 22:39:11.815463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:11.815430    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:39:11.816537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:11.816510    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:39:11.816680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:11.816557    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 22:39:12.237308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:12.237268    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:39:12.758245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:12.757344    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:39:12.758245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:12.757408    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 22:39:12.821789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:12.821751    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:39:12.822135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:12.822089    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 22:39:13.751179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:13.750745    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:39:13.751179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:13.750806    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 22:39:13.817253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:13.816671    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:39:13.817253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:13.816736    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 22:39:14.753114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:14.753078    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:39:14.753764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:14.753726    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 22:39:14.825906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:14.825870    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:39:14.826297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:14.826260    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 22:39:15.750608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:15.750570    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:39:15.750893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:15.750866    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 22:39:15.816384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:15.816343    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:39:15.816876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:15.816397    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 22:39:16.784612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:16.784572    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:39:16.786445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:16.786379    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:39:16.815684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:16.815652    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:39:16.815855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:16.815704    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 22:39:17.816090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:17.816050    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:39:17.816621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:17.816115    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 22:39:18.821669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:18.821635    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:39:18.822846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:18.822821    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:39:39.807149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:39.807107    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"]
Oct 30 22:39:39.807825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:39.807393    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy" containerID="cri-o://93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8" gracePeriod=30
Oct 30 22:39:40.819954 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:39:40.819911    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:39:45.145742 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.145714    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:39:45.161591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.161562    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-istio-token\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-istio-token\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.161758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.161610    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-certs\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.161758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.161686    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ccfd601-6d98-48b5-85e3-101586b02704-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccfd601-6d98-48b5-85e3-101586b02704-istio-podinfo\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.161758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.161745    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ccfd601-6d98-48b5-85e3-101586b02704-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccfd601-6d98-48b5-85e3-101586b02704-istiod-ca-cert\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.162004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.161984    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-credential-socket\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.162004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.161991    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-certs" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.162145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.162027    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-kube-api-access-vd5n8\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-kube-api-access-vd5n8\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.162145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.162091    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-envoy\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.162284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.162199    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-data\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.162284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.162244    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-credential-socket" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.162284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.162258    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccfd601-6d98-48b5-85e3-101586b02704-istiod-ca-cert" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:45.162284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.162270    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-socket\") pod \"1ccfd601-6d98-48b5-85e3-101586b02704\" (UID: \"1ccfd601-6d98-48b5-85e3-101586b02704\") "
Oct 30 22:39:45.162510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.162470    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-data" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.162598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.162577    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-socket" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.164151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.163582    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.164151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.163607    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccfd601-6d98-48b5-85e3-101586b02704-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.164151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.163620    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.164151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.163632    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.164151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.163645    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.164879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.164841    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-kube-api-access-vd5n8" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "kube-api-access-vd5n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:45.164965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.164940    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ccfd601-6d98-48b5-85e3-101586b02704-istio-podinfo" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:39:45.165063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.165040    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-istio-token" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:45.165177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.165156    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-envoy" pod "1ccfd601-6d98-48b5-85e3-101586b02704" (UID: "1ccfd601-6d98-48b5-85e3-101586b02704"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.265735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.265639    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.265735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.265680    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccfd601-6d98-48b5-85e3-101586b02704-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.265735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.265694    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd5n8\" (UniqueName: \"kubernetes.io/projected/1ccfd601-6d98-48b5-85e3-101586b02704-kube-api-access-vd5n8\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.265735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.265709    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccfd601-6d98-48b5-85e3-101586b02704-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:39:45.587096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.587000    4581 generic.go:356] "Generic (PLEG): container finished" podID="1ccfd601-6d98-48b5-85e3-101586b02704" containerID="93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8" exitCode=0
Oct 30 22:39:45.587096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.587063    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"
Oct 30 22:39:45.587096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.587071    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" event={"ID":"1ccfd601-6d98-48b5-85e3-101586b02704","Type":"ContainerDied","Data":"93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8"}
Oct 30 22:39:45.587450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.587139    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72" event={"ID":"1ccfd601-6d98-48b5-85e3-101586b02704","Type":"ContainerDied","Data":"327d5690f47ca8c143ac98b7c23d0af8ebded3164e2ea9d7703c2020743a659a"}
Oct 30 22:39:45.587450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.587171    4581 scope.go:122] "RemoveContainer" containerID="93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8"
Oct 30 22:39:45.606846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.606823    4581 scope.go:122] "RemoveContainer" containerID="62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4"
Oct 30 22:39:45.618678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.618647    4581 scope.go:122] "RemoveContainer" containerID="93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8"
Oct 30 22:39:45.618971 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:39:45.618950    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8\": container with ID starting with 93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8 not found: ID does not exist" containerID="93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8"
Oct 30 22:39:45.619059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.618986    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8"} err="failed to get container status \"93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8\": rpc error: code = NotFound desc = could not find container \"93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8\": container with ID starting with 93bb242934939f9e956f9ce04a87825fe2687ae9936eec31011d919f5327a1e8 not found: ID does not exist"
Oct 30 22:39:45.619059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.619011    4581 scope.go:122] "RemoveContainer" containerID="62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4"
Oct 30 22:39:45.619311 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:39:45.619287    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4\": container with ID starting with 62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4 not found: ID does not exist" containerID="62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4"
Oct 30 22:39:45.619366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.619317    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4"} err="failed to get container status \"62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4\": rpc error: code = NotFound desc = could not find container \"62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4\": container with ID starting with 62799a118584a5823e15ac035b12758cba74e8a2649bd3950f9dcef131db39c4 not found: ID does not exist"
Oct 30 22:39:45.640966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.640936    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"]
Oct 30 22:39:45.648196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:45.648165    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lnh72"]
Oct 30 22:39:46.231253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:46.228047    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" path="/var/lib/kubelet/pods/1ccfd601-6d98-48b5-85e3-101586b02704/volumes"
Oct 30 22:39:54.887074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:54.887026    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"]
Oct 30 22:39:54.887678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:54.887283    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy" containerID="cri-o://ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390" gracePeriod=30
Oct 30 22:39:57.726283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:57.726209    4581 generic.go:356] "Generic (PLEG): container finished" podID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerID="c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d" exitCode=0
Oct 30 22:39:57.726283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:57.726279    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" event={"ID":"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db","Type":"ContainerDied","Data":"c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d"}
Oct 30 22:39:57.726785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:57.726766    4581 scope.go:122] "RemoveContainer" containerID="c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d"
Oct 30 22:39:57.728907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:57.728873    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:57.729034 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:57.728986    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:57.729105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:57.729040    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:39:58.319382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:58.319332    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:39:58.749706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:58.749667    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" event={"ID":"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db","Type":"ContainerStarted","Data":"11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc"}
Oct 30 22:39:59.597663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:59.597631    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:39:59.598721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:59.598683    4581 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:39:59.598871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:59.598741    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:39:59.737412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:59.737378    4581 scope.go:122] "RemoveContainer" containerID="131fc586682c96740ee9a2b8c626f2734839869cc1482c2068d78159bfe3b1ee"
Oct 30 22:39:59.772633 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:59.771668    4581 scope.go:122] "RemoveContainer" containerID="075a7ca4706bef336f4147e51ec89b0bda3d9e5185c90541dc1eeb8dbfcecbee"
Oct 30 22:39:59.813672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:59.813646    4581 scope.go:122] "RemoveContainer" containerID="163202b471d2dacdf2ecbccbc2c30a0b7ed81508846fa1f872e46b7aab1dfc7e"
Oct 30 22:39:59.825717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:39:59.825688    4581 scope.go:122] "RemoveContainer" containerID="0394544b036e595088060280d3f6fd15d7b55812e98a12d3244f4e096a6d6329"
Oct 30 22:40:00.548056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.548020    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:40:00.607279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.606813    4581 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:00.607279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.606875    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.713935    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-token\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-token\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.714103    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-credential-socket\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.714161    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-socket\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.714251    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-data\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.714301    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-podinfo\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.714340    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-envoy\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.714377    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3b5e99-a183-44e8-a4ea-34113aac767b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3b5e99-a183-44e8-a4ea-34113aac767b-istiod-ca-cert\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.714463    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-certs\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.715614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.714541    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-kube-api-access-7q4sq\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-kube-api-access-7q4sq\") pod \"1f3b5e99-a183-44e8-a4ea-34113aac767b\" (UID: \"1f3b5e99-a183-44e8-a4ea-34113aac767b\") "
Oct 30 22:40:00.716859 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.716827    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-data" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:00.717094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.716839    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-credential-socket" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:00.717232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.717016    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-socket" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:00.717630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.717607    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3b5e99-a183-44e8-a4ea-34113aac767b-istiod-ca-cert" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:00.718908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.718878    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-kube-api-access-7q4sq" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "kube-api-access-7q4sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:00.719757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.719733    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-certs" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:00.722515 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.722486    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-envoy" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:00.722788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.722759    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-podinfo" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:00.722975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.722947    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-token" pod "1f3b5e99-a183-44e8-a4ea-34113aac767b" (UID: "1f3b5e99-a183-44e8-a4ea-34113aac767b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819386    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819422    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819436    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819449    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819463    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3b5e99-a183-44e8-a4ea-34113aac767b-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819476    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819491    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q4sq\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-kube-api-access-7q4sq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819504    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3b5e99-a183-44e8-a4ea-34113aac767b-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.822008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.819517    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3b5e99-a183-44e8-a4ea-34113aac767b-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:40:00.826304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.826269    4581 generic.go:356] "Generic (PLEG): container finished" podID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerID="ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390" exitCode=0
Oct 30 22:40:00.826628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.826603    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" event={"ID":"1f3b5e99-a183-44e8-a4ea-34113aac767b","Type":"ContainerDied","Data":"ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390"}
Oct 30 22:40:00.826760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.826744    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"
Oct 30 22:40:00.826828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.826815    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b" event={"ID":"1f3b5e99-a183-44e8-a4ea-34113aac767b","Type":"ContainerDied","Data":"d220aa80f809e27f03056857a44f5c0903e054a60b3ee9d15e9a46bc6c7f57f3"}
Oct 30 22:40:00.826911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.826903    4581 scope.go:122] "RemoveContainer" containerID="ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390"
Oct 30 22:40:00.846287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.846266    4581 scope.go:122] "RemoveContainer" containerID="ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390"
Oct 30 22:40:00.846648 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:40:00.846621    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390\": container with ID starting with ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390 not found: ID does not exist" containerID="ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390"
Oct 30 22:40:00.846786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.846657    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390"} err="failed to get container status \"ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390\": rpc error: code = NotFound desc = could not find container \"ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390\": container with ID starting with ec1e3807f40a8fd514508dd82b1c944878853b05486360eb50cd887a21221390 not found: ID does not exist"
Oct 30 22:40:00.919055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.919013    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"]
Oct 30 22:40:00.928645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:00.928608    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vbv8b"]
Oct 30 22:40:01.598089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:01.598051    4581 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:01.598346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:01.598114    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:02.100380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:02.100349    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" path="/var/lib/kubelet/pods/1f3b5e99-a183-44e8-a4ea-34113aac767b/volumes"
Oct 30 22:40:02.598399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:02.598357    4581 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:02.598594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:02.598421    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:03.598081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:03.598043    4581 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:03.598536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:03.598111    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:04.180203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.180168    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"]
Oct 30 22:40:04.181693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.181664    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy"
Oct 30 22:40:04.236088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.236057    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy"
Oct 30 22:40:04.236729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.236618    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy"
Oct 30 22:40:04.236878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.236864    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy"
Oct 30 22:40:04.237768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.237657    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3b5e99-a183-44e8-a4ea-34113aac767b" containerName="istio-proxy"
Oct 30 22:40:04.260276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.237925    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy"
Oct 30 22:40:04.260764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.260645    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy"
Oct 30 22:40:04.325198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.320983    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"]
Oct 30 22:40:04.325198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.321121    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.336065 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.336036    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-40229e\"/\"default-dockercfg-9gb2f\""
Oct 30 22:40:04.344878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.344844    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-40229e\"/\"test-model-40229e-kserve-self-signed-certs\""
Oct 30 22:40:04.345047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.344957    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-40229e\"/\"openshift-service-ca.crt\""
Oct 30 22:40:04.345047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.345007    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-40229e\"/\"kube-root-ca.crt\""
Oct 30 22:40:04.554401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.540837    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkgxs\" (UniqueName: \"kubernetes.io/projected/fcfc6689-f901-4959-aa31-1d11765ebc68-kube-api-access-kkgxs\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.554802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.554761    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-dshm\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.561437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.561384    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-home\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.587708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.587661    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfc6689-f901-4959-aa31-1d11765ebc68-tls-certs\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.588196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.588170    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-tmp-dir\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.588471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.588453    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-model-cache\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.613422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.610681    4581 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:04.613422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.610750    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:04.690671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.690628    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfc6689-f901-4959-aa31-1d11765ebc68-tls-certs\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.691107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.691081    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-tmp-dir\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.691381 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.691352    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-model-cache\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.691493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.691471    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkgxs\" (UniqueName: \"kubernetes.io/projected/fcfc6689-f901-4959-aa31-1d11765ebc68-kube-api-access-kkgxs\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.691553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.691522    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-dshm\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.691646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.691578    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-home\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.691646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.691619    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-tmp-dir\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.691952 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.691926    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-home\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.692385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.692362    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-model-cache\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.694445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.694420    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-dshm\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.695140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.695104    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfc6689-f901-4959-aa31-1d11765ebc68-tls-certs\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.713278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.710573    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkgxs\" (UniqueName: \"kubernetes.io/projected/fcfc6689-f901-4959-aa31-1d11765ebc68-kube-api-access-kkgxs\") pod \"test-model-40229e-kserve-66586c5b47-jfd7l\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") " pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:04.939373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:04.939333    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:05.348967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:05.348933    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"]
Oct 30 22:40:05.351453 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:40:05.351394    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcfc6689_f901_4959_aa31_1d11765ebc68.slice/crio-323cafba49730c47297263ebaf71df0da4cb20e30f2430e4fd32f11653d6bb37 WatchSource:0}: Error finding container 323cafba49730c47297263ebaf71df0da4cb20e30f2430e4fd32f11653d6bb37: Status 404 returned error can't find the container with id 323cafba49730c47297263ebaf71df0da4cb20e30f2430e4fd32f11653d6bb37
Oct 30 22:40:05.639052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:05.638663    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:40:05.648529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:05.641269    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:40:05.668327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:05.659500    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:40:05.991658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:05.991555    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l" event={"ID":"fcfc6689-f901-4959-aa31-1d11765ebc68","Type":"ContainerStarted","Data":"086fc814c6f80a471411f66d18d9be7bc0ab589d3c822d746bc7140b880af06b"}
Oct 30 22:40:05.991658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:05.991608    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l" event={"ID":"fcfc6689-f901-4959-aa31-1d11765ebc68","Type":"ContainerStarted","Data":"323cafba49730c47297263ebaf71df0da4cb20e30f2430e4fd32f11653d6bb37"}
Oct 30 22:40:06.032619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:06.032542    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l" podStartSLOduration=2.032520125 podStartE2EDuration="2.032520125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:06.024393456 +0000 UTC m=+1990.063332343" watchObservedRunningTime="2026-10-30 22:40:06.032520125 +0000 UTC m=+1990.071459013"
Oct 30 22:40:06.943245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:06.942370    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:10.854251 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:40:10.853410    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:40:14.287898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:14.287345    4581 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:40:14.287898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:14.287465    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:40:14.940330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:14.940284    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:14.957255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:14.957204    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:15.191579 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:15.191493    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:40:17.202187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:17.202154    4581 generic.go:356] "Generic (PLEG): container finished" podID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerID="798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c" exitCode=0
Oct 30 22:40:17.202187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:17.202191    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" event={"ID":"a451d097-a43a-42eb-beaa-e7b394e8efb9","Type":"ContainerDied","Data":"798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c"}
Oct 30 22:40:17.202761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:17.202706    4581 scope.go:122] "RemoveContainer" containerID="798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c"
Oct 30 22:40:17.204864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:17.204829    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:40:17.205002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:17.204924    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:40:17.205002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:17.204975    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:40:18.214606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:18.214570    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" event={"ID":"a451d097-a43a-42eb-beaa-e7b394e8efb9","Type":"ContainerStarted","Data":"7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6"}
Oct 30 22:40:18.515856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:18.515762    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:40:18.516035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:18.516013    4581 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:40:18.516092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:18.516060    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 22:40:19.516058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:19.516023    4581 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:40:19.516592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:19.516094    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 22:40:20.515852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:20.515811    4581 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:40:20.516049 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:20.515876    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 22:40:21.516472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:21.516436    4581 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:40:21.517155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:21.517087    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 22:40:22.515595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:22.515548    4581 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:40:22.515795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:22.515615    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 22:40:23.515954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:23.515917    4581 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:40:23.516395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:23.515972    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 22:40:24.523050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:24.523012    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:40:24.523607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:24.523517    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:40:24.524362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:24.524340    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:40:28.030871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.030829    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"]
Oct 30 22:40:28.032179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.032154    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy"
Oct 30 22:40:28.032179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.032178    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ccfd601-6d98-48b5-85e3-101586b02704" containerName="istio-proxy"
Oct 30 22:40:28.036560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.036530    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.049781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.049758    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-54c7f0-openshift-default-dockercfg-vj8jm\""
Oct 30 22:40:28.068267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.068201    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"]
Oct 30 22:40:28.174568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.174527    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.174788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.174694    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-certs\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.174788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.174735    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-istio-token\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.174930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.174815    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-envoy\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.174992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.174971    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-data\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.175110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.175081    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967c73d-9969-4099-a603-b69060bb3732-istiod-ca-cert\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.175187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.175170    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-credential-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.175395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.175377    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967c73d-9969-4099-a603-b69060bb3732-istio-podinfo\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.175465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.175408    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvnk9\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-kube-api-access-gvnk9\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277373    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967c73d-9969-4099-a603-b69060bb3732-istio-podinfo\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277427    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvnk9\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-kube-api-access-gvnk9\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277522    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277613    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-certs\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277643    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-istio-token\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277676    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-envoy\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277772    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-data\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277824    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967c73d-9969-4099-a603-b69060bb3732-istiod-ca-cert\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.277936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.277883    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-credential-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.278310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.278285    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-certs\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.278406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.278379    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-credential-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.278591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.278568    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-socket\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.278846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.278825    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-data\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.278958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.278942    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967c73d-9969-4099-a603-b69060bb3732-istiod-ca-cert\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.280700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.280674    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-envoy\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.280926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.280882    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967c73d-9969-4099-a603-b69060bb3732-istio-podinfo\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.298868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.298832    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvnk9\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-kube-api-access-gvnk9\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.299466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.299443    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-istio-token\") pod \"e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") " pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.352299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.352265    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:28.743120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.743091    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"]
Oct 30 22:40:28.744784 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:40:28.744756    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode967c73d_9969_4099_a603_b69060bb3732.slice/crio-9a81b5baecb47310990443bb06c804ca9e6b8c55bec11b24712387953529cff5 WatchSource:0}: Error finding container 9a81b5baecb47310990443bb06c804ca9e6b8c55bec11b24712387953529cff5: Status 404 returned error can't find the container with id 9a81b5baecb47310990443bb06c804ca9e6b8c55bec11b24712387953529cff5
Oct 30 22:40:28.748628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.748590    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:40:28.748750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.748682    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:40:28.748750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:28.748730    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:40:29.362274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:29.359417    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" event={"ID":"e967c73d-9969-4099-a603-b69060bb3732","Type":"ContainerStarted","Data":"e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c"}
Oct 30 22:40:29.362274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:29.359479    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" event={"ID":"e967c73d-9969-4099-a603-b69060bb3732","Type":"ContainerStarted","Data":"9a81b5baecb47310990443bb06c804ca9e6b8c55bec11b24712387953529cff5"}
Oct 30 22:40:29.413105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:29.413044    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" podStartSLOduration=1.413027007 podStartE2EDuration="1.413027007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:29.405512482 +0000 UTC m=+2013.444451367" watchObservedRunningTime="2026-10-30 22:40:29.413027007 +0000 UTC m=+2013.451965892"
Oct 30 22:40:30.373241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:30.372703    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:30.387259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:30.386693    4581 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:40:30.387259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:30.386766    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:40:31.374244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.373673    4581 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:40:31.374244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.373747    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:40:31.595140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.595107    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"]
Oct 30 22:40:31.601684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.601655    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.618611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.618018    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"]
Oct 30 22:40:31.623000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.622970    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c13db0\"/\"default-dockercfg-4nzgl\""
Oct 30 22:40:31.623598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.623573    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c13db0\"/\"test-model-c13db0-kserve-self-signed-certs\""
Oct 30 22:40:31.626056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.625992    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c13db0\"/\"kube-root-ca.crt\""
Oct 30 22:40:31.626161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.626057    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c13db0\"/\"openshift-service-ca.crt\""
Oct 30 22:40:31.698242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.697418    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-dshm\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.698242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.697507    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smhzx\" (UniqueName: \"kubernetes.io/projected/b1322a3c-9054-4605-987a-1fd1fb53f46b-kube-api-access-smhzx\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.698242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.697595    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-tmp-dir\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.698242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.697620    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1322a3c-9054-4605-987a-1fd1fb53f46b-tls-certs\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.698242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.697682    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-model-cache\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.698242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.697723    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-home\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.798882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.798835    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-dshm\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.799097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.798927    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smhzx\" (UniqueName: \"kubernetes.io/projected/b1322a3c-9054-4605-987a-1fd1fb53f46b-kube-api-access-smhzx\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.799097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.798999    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-tmp-dir\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.799097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.799019    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1322a3c-9054-4605-987a-1fd1fb53f46b-tls-certs\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.799097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.799062    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-model-cache\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.799341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.799100    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-home\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.799755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.799725    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-tmp-dir\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.800143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.799850    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-model-cache\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.800143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.800101    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-home\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.802290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.801969    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-dshm\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.802478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.802344    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1322a3c-9054-4605-987a-1fd1fb53f46b-tls-certs\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.846284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.837936    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smhzx\" (UniqueName: \"kubernetes.io/projected/b1322a3c-9054-4605-987a-1fd1fb53f46b-kube-api-access-smhzx\") pod \"test-model-c13db0-kserve-6f84486fd9-hxkbr\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") " pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:31.932973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:31.931620    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:32.375693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:32.375645    4581 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:40:32.376319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:32.375717    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:40:32.490826 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:40:32.490787    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1322a3c_9054_4605_987a_1fd1fb53f46b.slice/crio-fd27dd7fc822390c659458a65ae4a11d910732f0778be9f6c7b95f2b3def06af WatchSource:0}: Error finding container fd27dd7fc822390c659458a65ae4a11d910732f0778be9f6c7b95f2b3def06af: Status 404 returned error can't find the container with id fd27dd7fc822390c659458a65ae4a11d910732f0778be9f6c7b95f2b3def06af
Oct 30 22:40:32.491230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:32.491177    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"]
Oct 30 22:40:33.365257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:33.354611    4581 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:40:33.365257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:33.354672    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:40:33.493242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:33.491493    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr" event={"ID":"b1322a3c-9054-4605-987a-1fd1fb53f46b","Type":"ContainerStarted","Data":"a21ed1dfce311ccb68dcb74a7171d6e0f9269721c30b6704a586c1b71f780b87"}
Oct 30 22:40:33.493242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:33.491546    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr" event={"ID":"b1322a3c-9054-4605-987a-1fd1fb53f46b","Type":"ContainerStarted","Data":"fd27dd7fc822390c659458a65ae4a11d910732f0778be9f6c7b95f2b3def06af"}
Oct 30 22:40:33.545241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:33.544773    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr" podStartSLOduration=2.54475425 podStartE2EDuration="2.54475425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:33.542370183 +0000 UTC m=+2017.581309070" watchObservedRunningTime="2026-10-30 22:40:33.54475425 +0000 UTC m=+2017.583693137"
Oct 30 22:40:33.934477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:33.934445    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:34.355602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:34.355566    4581 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:40:34.355800 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:34.355628    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:40:35.380441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:35.380403    4581 patch_prober.go:29] interesting pod/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:40:35.381109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:35.381071    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:40:36.359755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:36.359669    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:36.360287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:36.360209    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:36.361230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:36.361197    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:40:40.824260 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:40:40.824182    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:40:41.934429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:41.934392    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:41.949952 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:41.949925    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:42.612671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:42.612640    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:40:56.187674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:56.187641    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"]
Oct 30 22:40:56.189068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:56.187860    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" containerID="cri-o://b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7" gracePeriod=30
Oct 30 22:40:59.997569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:40:59.997528    4581 scope.go:122] "RemoveContainer" containerID="20f14a5373b34053f0620d65ddcd08cbfb3615366d1dde2f1195a064f4363d4d"
Oct 30 22:41:00.009768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:00.009739    4581 scope.go:122] "RemoveContainer" containerID="1c327b244f844bb66c729d586daf4880cb3e1ed6ca73da0bf40fd3bd47e7b4ce"
Oct 30 22:41:00.021303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:00.021279    4581 scope.go:122] "RemoveContainer" containerID="1c4c6b2bbd43151c2b1518a3ed8dc879cc318f58333cd50ab418117d327f85b6"
Oct 30 22:41:01.313638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.313604    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:41:01.314011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.313683    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 22:41:01.530578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.530552    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:41:01.573083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.573044    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-kube-api-access-dcsrc\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-kube-api-access-dcsrc\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.573404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.573333    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-socket\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.573404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.573389    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-data\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.573600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.573430    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istiod-ca-cert\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.573600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.573507    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-envoy\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.573763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.573640    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-podinfo\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.573763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.573680    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-certs\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.573916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.573765    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-token\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-token\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.574046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.574021    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-credential-socket\") pod \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\" (UID: \"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13\") "
Oct 30 22:41:01.574376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.574347    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-socket" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.574624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.574602    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-data" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.575424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.575395    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.575653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.575429    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.575846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.575651    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-credential-socket" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.576299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.576273    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-certs" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.577807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.576660    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istiod-ca-cert" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:01.580164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.579681    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-token" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:01.580164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.579906    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-envoy" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:01.581866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.581840    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-podinfo" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:01.582140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.582123    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-kube-api-access-dcsrc" pod "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" (UID: "99b8345e-b2e7-48e1-a0f3-b02bf0e0af13"). InnerVolumeSpecName "kube-api-access-dcsrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:01.692175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.692133    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.692175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.692177    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.692444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.692191    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.692444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.692204    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.692444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.692237    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcsrc\" (UniqueName: \"kubernetes.io/projected/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-kube-api-access-dcsrc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.692444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.692250    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.692444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.692261    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:01.820932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.820840    4581 generic.go:356] "Generic (PLEG): container finished" podID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerID="b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7" exitCode=0
Oct 30 22:41:01.820932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.820893    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"
Oct 30 22:41:01.820932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.820920    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" event={"ID":"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13","Type":"ContainerDied","Data":"b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7"}
Oct 30 22:41:01.821206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.820979    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq" event={"ID":"99b8345e-b2e7-48e1-a0f3-b02bf0e0af13","Type":"ContainerDied","Data":"b27c61f2ad142b1ee9940d7c01ff1b92400a4d5cd717e1090fab303d1312290b"}
Oct 30 22:41:01.821206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.821007    4581 scope.go:122] "RemoveContainer" containerID="b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7"
Oct 30 22:41:01.835205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.835176    4581 scope.go:122] "RemoveContainer" containerID="b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7"
Oct 30 22:41:01.835651 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:41:01.835629    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7\": container with ID starting with b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7 not found: ID does not exist" containerID="b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7"
Oct 30 22:41:01.835739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.835662    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7"} err="failed to get container status \"b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7\": rpc error: code = NotFound desc = could not find container \"b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7\": container with ID starting with b3164865f05b9a7f1e0dd2efef8eedc61e95ec2085cf2f3decd4015431098ca7 not found: ID does not exist"
Oct 30 22:41:01.870791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.870757    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"]
Oct 30 22:41:01.880164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:01.880136    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-v5lkq"]
Oct 30 22:41:02.111051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:02.108396    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" path="/var/lib/kubelet/pods/99b8345e-b2e7-48e1-a0f3-b02bf0e0af13/volumes"
Oct 30 22:41:10.826737 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:41:10.826694    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:41:20.957481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:20.957417    4581 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:41:20.958087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:20.957522    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:41:24.231531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:24.231449    4581 generic.go:356] "Generic (PLEG): container finished" podID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerID="7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f" exitCode=0
Oct 30 22:41:24.231531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:24.231508    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" event={"ID":"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2","Type":"ContainerDied","Data":"7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f"}
Oct 30 22:41:24.232071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:24.231959    4581 scope.go:122] "RemoveContainer" containerID="7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f"
Oct 30 22:41:24.234170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:24.234136    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:41:24.234316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:24.234231    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:41:24.234316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:24.234272    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:41:25.245616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:25.245577    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" event={"ID":"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2","Type":"ContainerStarted","Data":"f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c"}
Oct 30 22:41:26.026686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:26.026651    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:41:26.026887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:26.026848    4581 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 22:41:26.026943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:26.026921    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 22:41:27.026951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:27.026910    4581 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 22:41:27.027538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:27.026977    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 22:41:28.026791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:28.026750    4581 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 22:41:28.026983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:28.026861    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 22:41:29.027051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:29.027008    4581 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 22:41:29.027610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:29.027079    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 22:41:30.027310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:30.027277    4581 patch_prober.go:29] interesting pod/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 22:41:30.027737 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:30.027331    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 22:41:31.031864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:31.031831    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:41:31.032401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:31.032243    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:41:31.033058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:31.033029    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:41:32.875410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.875359    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"]
Oct 30 22:41:32.877424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.877394    4581 cpu_manager.go:414] "containerMap: removing container" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy"
Oct 30 22:41:32.877424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.877423    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy"
Oct 30 22:41:32.877988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.877968    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b8345e-b2e7-48e1-a0f3-b02bf0e0af13" containerName="istio-proxy"
Oct 30 22:41:32.888922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.888893    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.906061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.906033    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"]
Oct 30 22:41:32.908815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.908792    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f9d87b-openshift-default-dockercfg-b6gdf\""
Oct 30 22:41:32.950976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.950939    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-certs\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.951185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.951049    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-envoy\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.951185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.951099    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-credential-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.951185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.951121    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.951185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.951153    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-podinfo\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.951472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.951267    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d006e94-53e2-4e63-9b5e-1632efdc3194-istiod-ca-cert\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.951472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.951291    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-data\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.951472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.951345    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rkq\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-kube-api-access-82rkq\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:32.951472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:32.951371    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-token\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.052542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.052500    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d006e94-53e2-4e63-9b5e-1632efdc3194-istiod-ca-cert\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.052763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.052553    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-data\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.052763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.052640    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rkq\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-kube-api-access-82rkq\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.052763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.052682    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-token\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.052976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.052934    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-certs\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.052997    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-data\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.053093    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-envoy\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.053179    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-credential-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.053230    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.053288    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-podinfo\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.053297    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d006e94-53e2-4e63-9b5e-1632efdc3194-istiod-ca-cert\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.053297    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-certs\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.053534    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-credential-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.053643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.053563    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-socket\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.055896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.055868    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-envoy\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.055896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.055891    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-podinfo\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.073254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.073176    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rkq\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-kube-api-access-82rkq\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.074271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.074199    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-token\") pod \"e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") " pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.205380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.205249    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:33.542394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.542367    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"]
Oct 30 22:41:33.543040 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:41:33.543012    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d006e94_53e2_4e63_9b5e_1632efdc3194.slice/crio-b3d362fcc7921a33e100bc33259eaa07e3bd861f828fdbaeba3c5302da3ecf3b WatchSource:0}: Error finding container b3d362fcc7921a33e100bc33259eaa07e3bd861f828fdbaeba3c5302da3ecf3b: Status 404 returned error can't find the container with id b3d362fcc7921a33e100bc33259eaa07e3bd861f828fdbaeba3c5302da3ecf3b
Oct 30 22:41:33.546141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.546106    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:41:33.546233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.546179    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:41:33.546286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:33.546210    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:41:34.385924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:34.385880    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" event={"ID":"5d006e94-53e2-4e63-9b5e-1632efdc3194","Type":"ContainerStarted","Data":"190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8"}
Oct 30 22:41:34.386439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:34.385951    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" event={"ID":"5d006e94-53e2-4e63-9b5e-1632efdc3194","Type":"ContainerStarted","Data":"b3d362fcc7921a33e100bc33259eaa07e3bd861f828fdbaeba3c5302da3ecf3b"}
Oct 30 22:41:34.473153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:34.472447    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" podStartSLOduration=2.472420806 podStartE2EDuration="2.472420806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:34.455626543 +0000 UTC m=+2078.494565429" watchObservedRunningTime="2026-10-30 22:41:34.472420806 +0000 UTC m=+2078.511359693"
Oct 30 22:41:35.205453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:35.205408    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:35.207062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:35.207031    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 30 22:41:35.207235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:35.207090    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 30 22:41:36.206454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:36.206411    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 30 22:41:36.206977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:36.206474    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 30 22:41:37.206173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:37.206131    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 30 22:41:37.206411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:37.206209    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 30 22:41:38.206243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:38.205997    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 30 22:41:38.206243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:38.206079    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 30 22:41:39.206471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:39.206436    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 30 22:41:39.206887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:39.206490    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 30 22:41:40.217464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.217420    4581 patch_prober.go:29] interesting pod/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 30 22:41:40.218109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.218069    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 30 22:41:40.477718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.477626    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"]
Oct 30 22:41:40.483019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.482988    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.504932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.504888    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"]
Oct 30 22:41:40.505328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.505293    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:40.505481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.505348    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:41:40.505481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.505396    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:40.505481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.505423    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:40.564190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.564151    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.564470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.564449    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.564561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.564542    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtkm\" (UniqueName: \"kubernetes.io/projected/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-kube-api-access-jjtkm\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.666405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.666368    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.666653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.666448    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtkm\" (UniqueName: \"kubernetes.io/projected/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-kube-api-access-jjtkm\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.666740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.666662    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.667169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.667144    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.669446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.669414    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.684495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.684466    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtkm\" (UniqueName: \"kubernetes.io/projected/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-kube-api-access-jjtkm\") pod \"maas-api-revoke-sub-74ebb966-7vm5d\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.799536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:40.799444    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:40.837176 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:41:40.837124    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:41:41.133187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:41.133161    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"]
Oct 30 22:41:41.134689 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:41:41.134658    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c3e2bb_13ff_4296_9b52_24d50d414b6c.slice/crio-36e02518a2cc19243b826fa06a53c020d12635f1debfee82d1c1123ce04359df WatchSource:0}: Error finding container 36e02518a2cc19243b826fa06a53c020d12635f1debfee82d1c1123ce04359df: Status 404 returned error can't find the container with id 36e02518a2cc19243b826fa06a53c020d12635f1debfee82d1c1123ce04359df
Oct 30 22:41:41.211474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:41.211443    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:41.211898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:41.211874    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:41.212572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:41.212552    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:41:41.495922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:41.495886    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d" event={"ID":"e7c3e2bb-13ff-4296-9b52-24d50d414b6c","Type":"ContainerStarted","Data":"36e02518a2cc19243b826fa06a53c020d12635f1debfee82d1c1123ce04359df"}
Oct 30 22:41:42.505764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:42.505728    4581 generic.go:356] "Generic (PLEG): container finished" podID="e7c3e2bb-13ff-4296-9b52-24d50d414b6c" containerID="a9c8df3258ecf03eb07abecd88ce82b819f6fcda258c4bd3a400cbfcec048f5b" exitCode=0
Oct 30 22:41:42.506202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:42.505818    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d" event={"ID":"e7c3e2bb-13ff-4296-9b52-24d50d414b6c","Type":"ContainerDied","Data":"a9c8df3258ecf03eb07abecd88ce82b819f6fcda258c4bd3a400cbfcec048f5b"}
Oct 30 22:41:43.723725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.723698    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:43.740382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.740346    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-xks-ca\") pod \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") "
Oct 30 22:41:43.740625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.740603    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-openshift-ca\") pod \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") "
Oct 30 22:41:43.740805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.740787    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-kube-api-access-jjtkm\" (UniqueName: \"kubernetes.io/projected/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-kube-api-access-jjtkm\") pod \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\" (UID: \"e7c3e2bb-13ff-4296-9b52-24d50d414b6c\") "
Oct 30 22:41:43.741045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.741015    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-openshift-ca" pod "e7c3e2bb-13ff-4296-9b52-24d50d414b6c" (UID: "e7c3e2bb-13ff-4296-9b52-24d50d414b6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:43.741775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.741748    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:43.742888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.742863    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-xks-ca" pod "e7c3e2bb-13ff-4296-9b52-24d50d414b6c" (UID: "e7c3e2bb-13ff-4296-9b52-24d50d414b6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:43.743174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.743146    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-kube-api-access-jjtkm" pod "e7c3e2bb-13ff-4296-9b52-24d50d414b6c" (UID: "e7c3e2bb-13ff-4296-9b52-24d50d414b6c"). InnerVolumeSpecName "kube-api-access-jjtkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:43.849531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.849435    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtkm\" (UniqueName: \"kubernetes.io/projected/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-kube-api-access-jjtkm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:43.849531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:43.849477    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c3e2bb-13ff-4296-9b52-24d50d414b6c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:44.526736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:44.526642    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"
Oct 30 22:41:44.526736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:44.526673    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d" event={"ID":"e7c3e2bb-13ff-4296-9b52-24d50d414b6c","Type":"ContainerDied","Data":"36e02518a2cc19243b826fa06a53c020d12635f1debfee82d1c1123ce04359df"}
Oct 30 22:41:44.526736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:44.526732    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e02518a2cc19243b826fa06a53c020d12635f1debfee82d1c1123ce04359df"
Oct 30 22:41:52.586260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.586209    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"]
Oct 30 22:41:52.587193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.587171    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e7c3e2bb-13ff-4296-9b52-24d50d414b6c" containerName="revoke-keys"
Oct 30 22:41:52.587193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.587192    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c3e2bb-13ff-4296-9b52-24d50d414b6c" containerName="revoke-keys"
Oct 30 22:41:52.587461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.587447    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c3e2bb-13ff-4296-9b52-24d50d414b6c" containerName="revoke-keys"
Oct 30 22:41:52.594533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.593697    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.616306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.616276    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:52.616702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.616684    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:52.616949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.616934    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:52.617190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.617176    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:41:52.624296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.624269    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"]
Oct 30 22:41:52.670098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.670061    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdzq\" (UniqueName: \"kubernetes.io/projected/b7d63e0e-b770-410a-a0f7-da8483a1803d-kube-api-access-nxdzq\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.670340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.670137    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.670340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.670190    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.772277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.772243    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdzq\" (UniqueName: \"kubernetes.io/projected/b7d63e0e-b770-410a-a0f7-da8483a1803d-kube-api-access-nxdzq\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.772493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.772319    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.772493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.772369    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.772962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.772940    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.775005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.774983    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.789701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.789672    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdzq\" (UniqueName: \"kubernetes.io/projected/b7d63e0e-b770-410a-a0f7-da8483a1803d-kube-api-access-nxdzq\") pod \"maas-api-revoke-sub-a5cc7d52-4h7zt\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:52.937812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:52.937780    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:53.267713 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:41:53.267677    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d63e0e_b770_410a_a0f7_da8483a1803d.slice/crio-3ef282a5248298cd7432becef3830bb43beef4c1bbda808d750dfc02fc443e99 WatchSource:0}: Error finding container 3ef282a5248298cd7432becef3830bb43beef4c1bbda808d750dfc02fc443e99: Status 404 returned error can't find the container with id 3ef282a5248298cd7432becef3830bb43beef4c1bbda808d750dfc02fc443e99
Oct 30 22:41:53.269608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:53.269579    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"]
Oct 30 22:41:53.631133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:53.631098    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt" event={"ID":"b7d63e0e-b770-410a-a0f7-da8483a1803d","Type":"ContainerStarted","Data":"3ef282a5248298cd7432becef3830bb43beef4c1bbda808d750dfc02fc443e99"}
Oct 30 22:41:54.659941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:54.659909    4581 generic.go:356] "Generic (PLEG): container finished" podID="b7d63e0e-b770-410a-a0f7-da8483a1803d" containerID="5e49cee37478604bcee6ab1a86e57dd6cfe275d2a31404625cf70b0fac2975ff" exitCode=0
Oct 30 22:41:54.660478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:54.660012    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt" event={"ID":"b7d63e0e-b770-410a-a0f7-da8483a1803d","Type":"ContainerDied","Data":"5e49cee37478604bcee6ab1a86e57dd6cfe275d2a31404625cf70b0fac2975ff"}
Oct 30 22:41:55.890168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:55.890139    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:55.930769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:55.930381    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-xks-ca\") pod \"b7d63e0e-b770-410a-a0f7-da8483a1803d\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") "
Oct 30 22:41:55.930769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:55.930523    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d63e0e-b770-410a-a0f7-da8483a1803d-kube-api-access-nxdzq\" (UniqueName: \"kubernetes.io/projected/b7d63e0e-b770-410a-a0f7-da8483a1803d-kube-api-access-nxdzq\") pod \"b7d63e0e-b770-410a-a0f7-da8483a1803d\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") "
Oct 30 22:41:55.930769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:55.930598    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-openshift-ca\") pod \"b7d63e0e-b770-410a-a0f7-da8483a1803d\" (UID: \"b7d63e0e-b770-410a-a0f7-da8483a1803d\") "
Oct 30 22:41:55.945694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:55.945651    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-openshift-ca" pod "b7d63e0e-b770-410a-a0f7-da8483a1803d" (UID: "b7d63e0e-b770-410a-a0f7-da8483a1803d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:55.952209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:55.952167    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-xks-ca" pod "b7d63e0e-b770-410a-a0f7-da8483a1803d" (UID: "b7d63e0e-b770-410a-a0f7-da8483a1803d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:55.954290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:55.952697    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d63e0e-b770-410a-a0f7-da8483a1803d-kube-api-access-nxdzq" pod "b7d63e0e-b770-410a-a0f7-da8483a1803d" (UID: "b7d63e0e-b770-410a-a0f7-da8483a1803d"). InnerVolumeSpecName "kube-api-access-nxdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:56.033856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:56.033817    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:56.034020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:56.033876    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdzq\" (UniqueName: \"kubernetes.io/projected/b7d63e0e-b770-410a-a0f7-da8483a1803d-kube-api-access-nxdzq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:56.034020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:56.033887    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d63e0e-b770-410a-a0f7-da8483a1803d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:41:56.683476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:56.681755    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"
Oct 30 22:41:56.683476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:56.681783    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt" event={"ID":"b7d63e0e-b770-410a-a0f7-da8483a1803d","Type":"ContainerDied","Data":"3ef282a5248298cd7432becef3830bb43beef4c1bbda808d750dfc02fc443e99"}
Oct 30 22:41:56.683476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:56.681836    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ef282a5248298cd7432becef3830bb43beef4c1bbda808d750dfc02fc443e99"
Oct 30 22:41:57.099644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:57.099559    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:41:57.106124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:41:57.106102    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:42:00.076017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:00.075983    4581 scope.go:122] "RemoveContainer" containerID="dc416bd79dabee4318d27d83f2435b3bca0e98ecd70195ffa8a814edc3bdbd86"
Oct 30 22:42:00.090814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:00.090773    4581 scope.go:122] "RemoveContainer" containerID="30e907f65dbe8cbfa6ea6576cd93be83e6e7b71931df9cd804c6e97a33a24164"
Oct 30 22:42:10.819845 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:42:10.819754    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:42:15.529284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.529249    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"]
Oct 30 22:42:15.530124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.530108    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b7d63e0e-b770-410a-a0f7-da8483a1803d" containerName="revoke-keys"
Oct 30 22:42:15.530185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.530125    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d63e0e-b770-410a-a0f7-da8483a1803d" containerName="revoke-keys"
Oct 30 22:42:15.530315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.530303    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d63e0e-b770-410a-a0f7-da8483a1803d" containerName="revoke-keys"
Oct 30 22:42:15.534491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.534471    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.556257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.555333    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"]
Oct 30 22:42:15.557448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.557421    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:42:15.557448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.557439    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:15.557448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.557423    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:15.557731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.557496    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:15.709048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.709014    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvtq\" (UniqueName: \"kubernetes.io/projected/bfc9dfe8-6830-4043-a79f-eb1107f453f8-kube-api-access-nfvtq\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.709288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.709120    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.709288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.709282    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.810785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.810695    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvtq\" (UniqueName: \"kubernetes.io/projected/bfc9dfe8-6830-4043-a79f-eb1107f453f8-kube-api-access-nfvtq\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.810785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.810751    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.810999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.810817    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.811475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.811455    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.813375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.813354    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.828951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.828921    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvtq\" (UniqueName: \"kubernetes.io/projected/bfc9dfe8-6830-4043-a79f-eb1107f453f8-kube-api-access-nfvtq\") pod \"maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:15.851165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:15.851136    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:16.177324 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:42:16.177298    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc9dfe8_6830_4043_a79f_eb1107f453f8.slice/crio-7d3631e0bab284b449733c42fd5ab81ecc30ddef62bae63efff990e925933b40 WatchSource:0}: Error finding container 7d3631e0bab284b449733c42fd5ab81ecc30ddef62bae63efff990e925933b40: Status 404 returned error can't find the container with id 7d3631e0bab284b449733c42fd5ab81ecc30ddef62bae63efff990e925933b40
Oct 30 22:42:16.178229 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:16.178181    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"]
Oct 30 22:42:16.898337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:16.894745    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr" event={"ID":"bfc9dfe8-6830-4043-a79f-eb1107f453f8","Type":"ContainerStarted","Data":"18a6ad29f49652b3d1bea6254f90c4b94a4c63b48d9d5e297e13f4478d13c0c0"}
Oct 30 22:42:16.898337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:16.895304    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr" event={"ID":"bfc9dfe8-6830-4043-a79f-eb1107f453f8","Type":"ContainerStarted","Data":"7d3631e0bab284b449733c42fd5ab81ecc30ddef62bae63efff990e925933b40"}
Oct 30 22:42:16.936444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:16.936381    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr" podStartSLOduration=1.9363662160000001 podStartE2EDuration="1.936366216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:16.929543366 +0000 UTC m=+2120.968482252" watchObservedRunningTime="2026-10-30 22:42:16.936366216 +0000 UTC m=+2120.975305138"
Oct 30 22:42:18.021242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:17.993141    4581 generic.go:356] "Generic (PLEG): container finished" podID="bfc9dfe8-6830-4043-a79f-eb1107f453f8" containerID="18a6ad29f49652b3d1bea6254f90c4b94a4c63b48d9d5e297e13f4478d13c0c0" exitCode=0
Oct 30 22:42:18.021242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:17.993574    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr" event={"ID":"bfc9dfe8-6830-4043-a79f-eb1107f453f8","Type":"ContainerDied","Data":"18a6ad29f49652b3d1bea6254f90c4b94a4c63b48d9d5e297e13f4478d13c0c0"}
Oct 30 22:42:19.668363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:19.668315    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:42:19.804237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:19.801313    4581 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:42:19.956304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:19.952294    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:42:20.304119 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.304085    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:20.748534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.748499    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc9dfe8-6830-4043-a79f-eb1107f453f8-kube-api-access-nfvtq\" (UniqueName: \"kubernetes.io/projected/bfc9dfe8-6830-4043-a79f-eb1107f453f8-kube-api-access-nfvtq\") pod \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") "
Oct 30 22:42:20.757599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.757571    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-openshift-ca\") pod \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") "
Oct 30 22:42:20.758311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.758286    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-xks-ca\") pod \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\" (UID: \"bfc9dfe8-6830-4043-a79f-eb1107f453f8\") "
Oct 30 22:42:20.759705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.756795    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc9dfe8-6830-4043-a79f-eb1107f453f8-kube-api-access-nfvtq" pod "bfc9dfe8-6830-4043-a79f-eb1107f453f8" (UID: "bfc9dfe8-6830-4043-a79f-eb1107f453f8"). InnerVolumeSpecName "kube-api-access-nfvtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:20.759865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.758095    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-openshift-ca" pod "bfc9dfe8-6830-4043-a79f-eb1107f453f8" (UID: "bfc9dfe8-6830-4043-a79f-eb1107f453f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:20.787775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.787721    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-xks-ca" pod "bfc9dfe8-6830-4043-a79f-eb1107f453f8" (UID: "bfc9dfe8-6830-4043-a79f-eb1107f453f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:20.861464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.861427    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:20.861464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.861465    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvtq\" (UniqueName: \"kubernetes.io/projected/bfc9dfe8-6830-4043-a79f-eb1107f453f8-kube-api-access-nfvtq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:20.861464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:20.861479    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc9dfe8-6830-4043-a79f-eb1107f453f8-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:21.253628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:21.253593    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"
Oct 30 22:42:21.253829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:21.253632    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr" event={"ID":"bfc9dfe8-6830-4043-a79f-eb1107f453f8","Type":"ContainerDied","Data":"7d3631e0bab284b449733c42fd5ab81ecc30ddef62bae63efff990e925933b40"}
Oct 30 22:42:21.253829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:21.253691    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d3631e0bab284b449733c42fd5ab81ecc30ddef62bae63efff990e925933b40"
Oct 30 22:42:25.995645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:25.995607    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"]
Oct 30 22:42:28.730652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:28.730621    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"]
Oct 30 22:42:28.739339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:28.739308    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6922d3-7bxnr"]
Oct 30 22:42:30.102613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:30.102574    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc9dfe8-6830-4043-a79f-eb1107f453f8" path="/var/lib/kubelet/pods/bfc9dfe8-6830-4043-a79f-eb1107f453f8/volumes"
Oct 30 22:42:30.996588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:30.996541    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m" podUID="08725481-e08a-4302-a783-efd11a4a4338" containerName="payload-processing" containerID="cri-o://7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b" gracePeriod=25
Oct 30 22:42:36.200595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.200557    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"]
Oct 30 22:42:36.201140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.200773    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" containerID="cri-o://8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814" gracePeriod=30
Oct 30 22:42:36.213008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.212962    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"]
Oct 30 22:42:36.213315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.213259    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy" containerID="cri-o://a411e06106481a6badcef8a3da6bd000d01923e245685dde15024a1e747ac076" gracePeriod=30
Oct 30 22:42:36.219484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.219460    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"]
Oct 30 22:42:36.219699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.219678    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy" containerID="cri-o://a8702974b2d4b7ac99a46754f75cce1a5c409d0fb2d1eb36c7eafa8d2765f04e" gracePeriod=30
Oct 30 22:42:36.972102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.972065    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"]
Oct 30 22:42:36.973006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.972988    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bfc9dfe8-6830-4043-a79f-eb1107f453f8" containerName="revoke-keys"
Oct 30 22:42:36.973075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.973008    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc9dfe8-6830-4043-a79f-eb1107f453f8" containerName="revoke-keys"
Oct 30 22:42:36.973157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.973147    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc9dfe8-6830-4043-a79f-eb1107f453f8" containerName="revoke-keys"
Oct 30 22:42:36.978510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.978485    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:36.991541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.991522    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:36.999687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:36.999662    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"]
Oct 30 22:42:37.000532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.000514    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:37.000532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.000523    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:37.000857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.000839    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:42:37.075752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.075708    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.075975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.075792    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.076089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.076066    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrgc\" (UniqueName: \"kubernetes.io/projected/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-kube-api-access-wmrgc\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.184243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.177468    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.184243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.177819    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrgc\" (UniqueName: \"kubernetes.io/projected/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-kube-api-access-wmrgc\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.184243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.178050    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.184243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.178928    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.212831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.212797    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.230870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.230788    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrgc\" (UniqueName: \"kubernetes.io/projected/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-kube-api-access-wmrgc\") pod \"maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.297204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.297168    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:37.621886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.621855    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"]
Oct 30 22:42:37.622924 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:42:37.622897    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b1a190_38f2_4bf5_a66b_5c13bb5cee41.slice/crio-b894304ff1e98f2404dad410b6988d0473c44002c2bc88d6cd16c61b4a556054 WatchSource:0}: Error finding container b894304ff1e98f2404dad410b6988d0473c44002c2bc88d6cd16c61b4a556054: Status 404 returned error can't find the container with id b894304ff1e98f2404dad410b6988d0473c44002c2bc88d6cd16c61b4a556054
Oct 30 22:42:37.970028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:37.969934    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m" podUID="08725481-e08a-4302-a783-efd11a4a4338" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:38.133521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.133486    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:38.133707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.133546    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:38.226631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.226541    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:38.226631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.226605    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:38.521243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.521147    4581 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:38.521422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.521204    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:38.579886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.579846    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd" event={"ID":"52b1a190-38f2-4bf5-a66b-5c13bb5cee41","Type":"ContainerStarted","Data":"686ea8d3b3343ae08cd6eb5897f9117b4f6f340dc26a3a2efb997868c704cfde"}
Oct 30 22:42:38.580085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.579914    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd" event={"ID":"52b1a190-38f2-4bf5-a66b-5c13bb5cee41","Type":"ContainerStarted","Data":"b894304ff1e98f2404dad410b6988d0473c44002c2bc88d6cd16c61b4a556054"}
Oct 30 22:42:38.617161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:38.617107    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd" podStartSLOduration=2.61709061 podStartE2EDuration="2.61709061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:38.614627448 +0000 UTC m=+2142.653566335" watchObservedRunningTime="2026-10-30 22:42:38.61709061 +0000 UTC m=+2142.656029495"
Oct 30 22:42:39.594720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:39.594682    4581 generic.go:356] "Generic (PLEG): container finished" podID="52b1a190-38f2-4bf5-a66b-5c13bb5cee41" containerID="686ea8d3b3343ae08cd6eb5897f9117b4f6f340dc26a3a2efb997868c704cfde" exitCode=0
Oct 30 22:42:39.595268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:39.594774    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd" event={"ID":"52b1a190-38f2-4bf5-a66b-5c13bb5cee41","Type":"ContainerDied","Data":"686ea8d3b3343ae08cd6eb5897f9117b4f6f340dc26a3a2efb997868c704cfde"}
Oct 30 22:42:40.824021 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:42:40.823987    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:42:40.838099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:40.838075    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:40.949616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:40.949511    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-openshift-ca\") pod \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") "
Oct 30 22:42:40.949806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:40.949632    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-xks-ca\") pod \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") "
Oct 30 22:42:40.949806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:40.949718    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-kube-api-access-wmrgc\" (UniqueName: \"kubernetes.io/projected/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-kube-api-access-wmrgc\") pod \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\" (UID: \"52b1a190-38f2-4bf5-a66b-5c13bb5cee41\") "
Oct 30 22:42:40.949943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:40.949893    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-openshift-ca" pod "52b1a190-38f2-4bf5-a66b-5c13bb5cee41" (UID: "52b1a190-38f2-4bf5-a66b-5c13bb5cee41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:40.950819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:40.950785    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:40.952766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:40.952741    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-xks-ca" pod "52b1a190-38f2-4bf5-a66b-5c13bb5cee41" (UID: "52b1a190-38f2-4bf5-a66b-5c13bb5cee41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:40.953045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:40.953016    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-kube-api-access-wmrgc" pod "52b1a190-38f2-4bf5-a66b-5c13bb5cee41" (UID: "52b1a190-38f2-4bf5-a66b-5c13bb5cee41"). InnerVolumeSpecName "kube-api-access-wmrgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.059239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.054744    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.059239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.054780    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrgc\" (UniqueName: \"kubernetes.io/projected/52b1a190-38f2-4bf5-a66b-5c13bb5cee41-kube-api-access-wmrgc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.505634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.505609    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.576259    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-data\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.576319    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-certs\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.576376    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-credential-socket\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.576418    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ac86a2-3a1f-43d8-b834-717d0415646e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ac86a2-3a1f-43d8-b834-717d0415646e-istiod-ca-cert\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.576555    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-envoy\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.576666    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-socket\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.576799    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-podinfo\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.576908    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-kube-api-access-8fhkb\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-kube-api-access-8fhkb\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.579241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.577044    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-token\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-token\") pod \"c0ac86a2-3a1f-43d8-b834-717d0415646e\" (UID: \"c0ac86a2-3a1f-43d8-b834-717d0415646e\") "
Oct 30 22:42:41.594280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.585906    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-certs" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.594280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.586906    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-credential-socket" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.594280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.587170    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-socket" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.594280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.588785    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ac86a2-3a1f-43d8-b834-717d0415646e-istiod-ca-cert" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.600353 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.600318    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-data" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.601276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.601252    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.601379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.601281    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.601379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.601295    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.601379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.601311    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ac86a2-3a1f-43d8-b834-717d0415646e-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.601379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.601326    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.604104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.601532    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-podinfo" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.604104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.602823    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-token" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.604104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.603374    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-envoy" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.623022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.622998    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"
Oct 30 22:42:41.627101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.623859    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd" event={"ID":"52b1a190-38f2-4bf5-a66b-5c13bb5cee41","Type":"ContainerDied","Data":"b894304ff1e98f2404dad410b6988d0473c44002c2bc88d6cd16c61b4a556054"}
Oct 30 22:42:41.627358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.627337    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b894304ff1e98f2404dad410b6988d0473c44002c2bc88d6cd16c61b4a556054"
Oct 30 22:42:41.628452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.628424    4581 generic.go:356] "Generic (PLEG): container finished" podID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerID="a8702974b2d4b7ac99a46754f75cce1a5c409d0fb2d1eb36c7eafa8d2765f04e" exitCode=0
Oct 30 22:42:41.628685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.628613    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" event={"ID":"df6cfbab-0eb3-4312-9eb1-9101cf84b148","Type":"ContainerDied","Data":"a8702974b2d4b7ac99a46754f75cce1a5c409d0fb2d1eb36c7eafa8d2765f04e"}
Oct 30 22:42:41.631096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.631051    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-kube-api-access-8fhkb" pod "c0ac86a2-3a1f-43d8-b834-717d0415646e" (UID: "c0ac86a2-3a1f-43d8-b834-717d0415646e"). InnerVolumeSpecName "kube-api-access-8fhkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.631298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.631123    4581 generic.go:356] "Generic (PLEG): container finished" podID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerID="8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814" exitCode=0
Oct 30 22:42:41.632662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.631441    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"
Oct 30 22:42:41.638003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.637968    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" event={"ID":"c0ac86a2-3a1f-43d8-b834-717d0415646e","Type":"ContainerDied","Data":"8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814"}
Oct 30 22:42:41.638165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.638037    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5" event={"ID":"c0ac86a2-3a1f-43d8-b834-717d0415646e","Type":"ContainerDied","Data":"2f7e874fd0541b0f91e11bcaed3fc527a035bd27dc8996a4b72af5367b9a00ab"}
Oct 30 22:42:41.638335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.638260    4581 scope.go:122] "RemoveContainer" containerID="8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814"
Oct 30 22:42:41.663523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.651416    4581 generic.go:356] "Generic (PLEG): container finished" podID="caef3b1d-a680-4780-ac73-4b41f033d942" containerID="a411e06106481a6badcef8a3da6bd000d01923e245685dde15024a1e747ac076" exitCode=0
Oct 30 22:42:41.663523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.651525    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" event={"ID":"caef3b1d-a680-4780-ac73-4b41f033d942","Type":"ContainerDied","Data":"a411e06106481a6badcef8a3da6bd000d01923e245685dde15024a1e747ac076"}
Oct 30 22:42:41.679152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.679125    4581 scope.go:122] "RemoveContainer" containerID="8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814"
Oct 30 22:42:41.680060 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:42:41.679817    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814\": container with ID starting with 8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814 not found: ID does not exist" containerID="8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814"
Oct 30 22:42:41.680060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.679866    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814"} err="failed to get container status \"8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814\": rpc error: code = NotFound desc = could not find container \"8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814\": container with ID starting with 8624781025c1121de4ec83d877791bdcfd712bf49b9e3e1b17f449cde4d10814 not found: ID does not exist"
Oct 30 22:42:41.688320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.688294    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:42:41.690111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.690088    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:42:41.705592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.705566    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.705829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.705793    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fhkb\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-kube-api-access-8fhkb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.705829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.705826    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.705990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.705842    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ac86a2-3a1f-43d8-b834-717d0415646e-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.715765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.715337    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"]
Oct 30 22:42:41.721314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.721289    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-tjbw5"]
Oct 30 22:42:41.807055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.806977    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-istio-token\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-istio-token\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.807055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807025    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-data\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.807352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807108    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-socket\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.807352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807186    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-envoy\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.807473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807383    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-certs\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.807473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807399    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-data" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.807473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807406    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-socket" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.807473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807460    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-kube-api-access-ss95j\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-kube-api-access-ss95j\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.807692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807510    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-credential-socket\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.807692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807657    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-certs" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.807804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807725    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-token\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-token\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.807804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807748    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-credential-socket" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.807929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807814    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istiod-ca-cert\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.807929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807843    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-podinfo\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.807929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.807910    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caef3b1d-a680-4780-ac73-4b41f033d942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef3b1d-a680-4780-ac73-4b41f033d942-istio-podinfo\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.808165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808146    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-credential-socket\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.808263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808193    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-certs\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.808342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808286    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caef3b1d-a680-4780-ac73-4b41f033d942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef3b1d-a680-4780-ac73-4b41f033d942-istiod-ca-cert\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.808342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808335    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-envoy\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.808625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808357    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istiod-ca-cert" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.808625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808395    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-socket\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.808625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808450    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-data\") pod \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\" (UID: \"df6cfbab-0eb3-4312-9eb1-9101cf84b148\") "
Oct 30 22:42:41.808625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808530    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-kube-api-access-c2q82\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-kube-api-access-c2q82\") pod \"caef3b1d-a680-4780-ac73-4b41f033d942\" (UID: \"caef3b1d-a680-4780-ac73-4b41f033d942\") "
Oct 30 22:42:41.808625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.808597    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-credential-socket" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.809574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.809548    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.809693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.809582    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.809693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.809598    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.809693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.809613    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.809693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.809628    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.809693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.809642    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.810132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.810109    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-istio-token" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.811108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.810318    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-envoy" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.811108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.810447    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-certs" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.811108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.810643    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-socket" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.811108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.810646    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-kube-api-access-ss95j" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "kube-api-access-ss95j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.811108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.810685    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caef3b1d-a680-4780-ac73-4b41f033d942-istiod-ca-cert" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.811108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.810815    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-data" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.820085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.819751    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-podinfo" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.820085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.819861    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-token" pod "df6cfbab-0eb3-4312-9eb1-9101cf84b148" (UID: "df6cfbab-0eb3-4312-9eb1-9101cf84b148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.820085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.819871    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-envoy" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.820530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.820373    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caef3b1d-a680-4780-ac73-4b41f033d942-istio-podinfo" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.820530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.820489    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-kube-api-access-c2q82" pod "caef3b1d-a680-4780-ac73-4b41f033d942" (UID: "caef3b1d-a680-4780-ac73-4b41f033d942"). InnerVolumeSpecName "kube-api-access-c2q82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.911600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911561    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.911600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911596    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss95j\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-kube-api-access-ss95j\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.911600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911607    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911615    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911630    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef3b1d-a680-4780-ac73-4b41f033d942-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911641    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911650    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef3b1d-a680-4780-ac73-4b41f033d942-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911658    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911666    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef3b1d-a680-4780-ac73-4b41f033d942-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911673    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6cfbab-0eb3-4312-9eb1-9101cf84b148-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911683    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2q82\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-kube-api-access-c2q82\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.912054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:41.911693    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caef3b1d-a680-4780-ac73-4b41f033d942-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:42.100931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.100855    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" path="/var/lib/kubelet/pods/c0ac86a2-3a1f-43d8-b834-717d0415646e/volumes"
Oct 30 22:42:42.664781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.664735    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw" event={"ID":"df6cfbab-0eb3-4312-9eb1-9101cf84b148","Type":"ContainerDied","Data":"be5b9e64eb3926a1895f08593de1baa97098786dd5be262c9d75c2eb49d281d3"}
Oct 30 22:42:42.664994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.664847    4581 scope.go:122] "RemoveContainer" containerID="a8702974b2d4b7ac99a46754f75cce1a5c409d0fb2d1eb36c7eafa8d2765f04e"
Oct 30 22:42:42.665062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.665008    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"
Oct 30 22:42:42.670152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.670117    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544" event={"ID":"caef3b1d-a680-4780-ac73-4b41f033d942","Type":"ContainerDied","Data":"a75d4a0dc4384d75cc9e00e98bf2e812b8ceecf8cf9ae14645bc4bffdd900b65"}
Oct 30 22:42:42.670309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.670138    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"
Oct 30 22:42:42.680205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.680183    4581 scope.go:122] "RemoveContainer" containerID="a411e06106481a6badcef8a3da6bd000d01923e245685dde15024a1e747ac076"
Oct 30 22:42:42.716733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.716705    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"]
Oct 30 22:42:42.725455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.725428    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-df6cw"]
Oct 30 22:42:42.759205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.759169    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"]
Oct 30 22:42:42.765020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:42.764995    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-6t544"]
Oct 30 22:42:44.103063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.103029    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" path="/var/lib/kubelet/pods/caef3b1d-a680-4780-ac73-4b41f033d942/volumes"
Oct 30 22:42:44.103625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.103609    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" path="/var/lib/kubelet/pods/df6cfbab-0eb3-4312-9eb1-9101cf84b148/volumes"
Oct 30 22:42:44.923278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.923240    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"]
Oct 30 22:42:44.924057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924042    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy"
Oct 30 22:42:44.924057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924057    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy"
Oct 30 22:42:44.924206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924084    4581 cpu_manager.go:414] "containerMap: removing container" podUID="52b1a190-38f2-4bf5-a66b-5c13bb5cee41" containerName="revoke-keys"
Oct 30 22:42:44.924206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924089    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b1a190-38f2-4bf5-a66b-5c13bb5cee41" containerName="revoke-keys"
Oct 30 22:42:44.924206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924110    4581 cpu_manager.go:414] "containerMap: removing container" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy"
Oct 30 22:42:44.924206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924115    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy"
Oct 30 22:42:44.924206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924129    4581 cpu_manager.go:414] "containerMap: removing container" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy"
Oct 30 22:42:44.924206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924134    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy"
Oct 30 22:42:44.924587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924270    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="caef3b1d-a680-4780-ac73-4b41f033d942" containerName="istio-proxy"
Oct 30 22:42:44.924587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924282    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="df6cfbab-0eb3-4312-9eb1-9101cf84b148" containerName="istio-proxy"
Oct 30 22:42:44.924587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924295    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ac86a2-3a1f-43d8-b834-717d0415646e" containerName="istio-proxy"
Oct 30 22:42:44.924587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.924304    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b1a190-38f2-4bf5-a66b-5c13bb5cee41" containerName="revoke-keys"
Oct 30 22:42:44.929799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.929775    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:44.951970 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.951917    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:44.952116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.951992    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:44.955887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.955865    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:44.955991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.955899    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:42:44.962484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.962460    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"]
Oct 30 22:42:44.987655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.987620    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:44.987931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.987909    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:44.988030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:44.988007    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq9j7\" (UniqueName: \"kubernetes.io/projected/023343d0-b093-49e2-9a63-d6e9922940cc-kube-api-access-lq9j7\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:45.090079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.090039    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:45.090307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.090285    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:45.090389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.090349    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq9j7\" (UniqueName: \"kubernetes.io/projected/023343d0-b093-49e2-9a63-d6e9922940cc-kube-api-access-lq9j7\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:45.090889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.090862    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:45.092983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.092960    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:45.107730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.107699    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq9j7\" (UniqueName: \"kubernetes.io/projected/023343d0-b093-49e2-9a63-d6e9922940cc-kube-api-access-lq9j7\") pod \"maas-api-revoke-sub-6c2884a3-jfq4b\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:45.245081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.244997    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:45.572738 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:42:45.572693    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023343d0_b093_49e2_9a63_d6e9922940cc.slice/crio-d4292882594dd7ef7675337fc948946e96bd28490363355722a63615173cd889 WatchSource:0}: Error finding container d4292882594dd7ef7675337fc948946e96bd28490363355722a63615173cd889: Status 404 returned error can't find the container with id d4292882594dd7ef7675337fc948946e96bd28490363355722a63615173cd889
Oct 30 22:42:45.572738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.572727    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"]
Oct 30 22:42:45.735270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:45.735207    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b" event={"ID":"023343d0-b093-49e2-9a63-d6e9922940cc","Type":"ContainerStarted","Data":"d4292882594dd7ef7675337fc948946e96bd28490363355722a63615173cd889"}
Oct 30 22:42:46.751955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:46.751922    4581 generic.go:356] "Generic (PLEG): container finished" podID="023343d0-b093-49e2-9a63-d6e9922940cc" containerID="eb9d1d544a66c24e7672a6456980702cf62a8ed1cfe9f283eb7ae6ceb86b554d" exitCode=0
Oct 30 22:42:46.752515 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:46.752012    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b" event={"ID":"023343d0-b093-49e2-9a63-d6e9922940cc","Type":"ContainerDied","Data":"eb9d1d544a66c24e7672a6456980702cf62a8ed1cfe9f283eb7ae6ceb86b554d"}
Oct 30 22:42:47.452089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:47.452046    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"]
Oct 30 22:42:47.949407 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:47.949382    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:47.969871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:47.969837    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m" podUID="08725481-e08a-4302-a783-efd11a4a4338" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:48.110303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.072985    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023343d0-b093-49e2-9a63-d6e9922940cc-kube-api-access-lq9j7\" (UniqueName: \"kubernetes.io/projected/023343d0-b093-49e2-9a63-d6e9922940cc-kube-api-access-lq9j7\") pod \"023343d0-b093-49e2-9a63-d6e9922940cc\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") "
Oct 30 22:42:48.110303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.073081    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-openshift-ca\") pod \"023343d0-b093-49e2-9a63-d6e9922940cc\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") "
Oct 30 22:42:48.110303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.073254    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-xks-ca\") pod \"023343d0-b093-49e2-9a63-d6e9922940cc\" (UID: \"023343d0-b093-49e2-9a63-d6e9922940cc\") "
Oct 30 22:42:48.154866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.153459    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-openshift-ca" pod "023343d0-b093-49e2-9a63-d6e9922940cc" (UID: "023343d0-b093-49e2-9a63-d6e9922940cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:48.215704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.215647    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023343d0-b093-49e2-9a63-d6e9922940cc-kube-api-access-lq9j7" pod "023343d0-b093-49e2-9a63-d6e9922940cc" (UID: "023343d0-b093-49e2-9a63-d6e9922940cc"). InnerVolumeSpecName "kube-api-access-lq9j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:48.249240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.245445    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-xks-ca" pod "023343d0-b093-49e2-9a63-d6e9922940cc" (UID: "023343d0-b093-49e2-9a63-d6e9922940cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:48.307727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.307593    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:48.415471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.415246    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023343d0-b093-49e2-9a63-d6e9922940cc-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:48.415706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.415685    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq9j7\" (UniqueName: \"kubernetes.io/projected/023343d0-b093-49e2-9a63-d6e9922940cc-kube-api-access-lq9j7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:48.785906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.785810    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b" event={"ID":"023343d0-b093-49e2-9a63-d6e9922940cc","Type":"ContainerDied","Data":"d4292882594dd7ef7675337fc948946e96bd28490363355722a63615173cd889"}
Oct 30 22:42:48.786295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.786272    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4292882594dd7ef7675337fc948946e96bd28490363355722a63615173cd889"
Oct 30 22:42:48.786456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:48.785963    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"
Oct 30 22:42:51.311356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.311324    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:42:51.361074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.361038    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08725481-e08a-4302-a783-efd11a4a4338-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725481-e08a-4302-a783-efd11a4a4338-plugins-config-volume\") pod \"08725481-e08a-4302-a783-efd11a4a4338\" (UID: \"08725481-e08a-4302-a783-efd11a4a4338\") "
Oct 30 22:42:51.361311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.361091    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08725481-e08a-4302-a783-efd11a4a4338-kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/08725481-e08a-4302-a783-efd11a4a4338-kube-api-access-hh84x\") pod \"08725481-e08a-4302-a783-efd11a4a4338\" (UID: \"08725481-e08a-4302-a783-efd11a4a4338\") "
Oct 30 22:42:51.362791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.362761    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08725481-e08a-4302-a783-efd11a4a4338-plugins-config-volume" pod "08725481-e08a-4302-a783-efd11a4a4338" (UID: "08725481-e08a-4302-a783-efd11a4a4338"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:51.363744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.363713    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08725481-e08a-4302-a783-efd11a4a4338-kube-api-access-hh84x" pod "08725481-e08a-4302-a783-efd11a4a4338" (UID: "08725481-e08a-4302-a783-efd11a4a4338"). InnerVolumeSpecName "kube-api-access-hh84x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:51.463468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.463434    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08725481-e08a-4302-a783-efd11a4a4338-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:51.463468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.463465    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/08725481-e08a-4302-a783-efd11a4a4338-kube-api-access-hh84x\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:42:51.818010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.817973    4581 generic.go:356] "Generic (PLEG): container finished" podID="08725481-e08a-4302-a783-efd11a4a4338" containerID="7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b" exitCode=0
Oct 30 22:42:51.818267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.818020    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"
Oct 30 22:42:51.818267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.818049    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m" event={"ID":"08725481-e08a-4302-a783-efd11a4a4338","Type":"ContainerDied","Data":"7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b"}
Oct 30 22:42:51.818267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.818111    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m" event={"ID":"08725481-e08a-4302-a783-efd11a4a4338","Type":"ContainerDied","Data":"9eef16a0182eb263a5eba666514cc21a9b69807e580d2004b601d8f7c00a740d"}
Oct 30 22:42:51.818267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.818143    4581 scope.go:122] "RemoveContainer" containerID="7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b"
Oct 30 22:42:51.832810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.832785    4581 scope.go:122] "RemoveContainer" containerID="7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b"
Oct 30 22:42:51.833258 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:42:51.833197    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b\": container with ID starting with 7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b not found: ID does not exist" containerID="7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b"
Oct 30 22:42:51.833380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.833251    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b"} err="failed to get container status \"7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b\": rpc error: code = NotFound desc = could not find container \"7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b\": container with ID starting with 7430ef0f31465f9278d12045fefec8fae45fee98678a59e62ba7c979cf9f216b not found: ID does not exist"
Oct 30 22:42:51.871029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.870996    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"]
Oct 30 22:42:51.876728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:51.876702    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6922d3-59cf97fb57-v5z2m"]
Oct 30 22:42:52.100275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:52.100152    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08725481-e08a-4302-a783-efd11a4a4338" path="/var/lib/kubelet/pods/08725481-e08a-4302-a783-efd11a4a4338/volumes"
Oct 30 22:42:52.452875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:52.452838    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs" podUID="27120312-1468-480d-9899-8768fd4593d3" containerName="payload-pre-processing" containerID="cri-o://2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e" gracePeriod=25
Oct 30 22:42:56.608139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.608101    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"]
Oct 30 22:42:56.608926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.608908    4581 cpu_manager.go:414] "containerMap: removing container" podUID="023343d0-b093-49e2-9a63-d6e9922940cc" containerName="revoke-keys"
Oct 30 22:42:56.608968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.608927    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023343d0-b093-49e2-9a63-d6e9922940cc" containerName="revoke-keys"
Oct 30 22:42:56.609005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.608972    4581 cpu_manager.go:414] "containerMap: removing container" podUID="08725481-e08a-4302-a783-efd11a4a4338" containerName="payload-processing"
Oct 30 22:42:56.609005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.608978    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08725481-e08a-4302-a783-efd11a4a4338" containerName="payload-processing"
Oct 30 22:42:56.609129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.609117    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="08725481-e08a-4302-a783-efd11a4a4338" containerName="payload-processing"
Oct 30 22:42:56.609167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.609136    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="023343d0-b093-49e2-9a63-d6e9922940cc" containerName="revoke-keys"
Oct 30 22:42:56.614467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.614444    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.634499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.634473    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"]
Oct 30 22:42:56.637616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.637588    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:56.637616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.637589    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:56.637790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.637612    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:56.637790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.637683    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:42:56.763058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.763029    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.763274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.763172    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mxb\" (UniqueName: \"kubernetes.io/projected/7fff5de9-809a-446c-aa19-79020a584b4a-kube-api-access-r7mxb\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.763371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.763353    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.865024 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.864936    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.865168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.865079    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mxb\" (UniqueName: \"kubernetes.io/projected/7fff5de9-809a-446c-aa19-79020a584b4a-kube-api-access-r7mxb\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.865261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.865245    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.866109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.866087    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.868842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.868814    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.887202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.887163    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mxb\" (UniqueName: \"kubernetes.io/projected/7fff5de9-809a-446c-aa19-79020a584b4a-kube-api-access-r7mxb\") pod \"maas-api-revoke-sub-e55e2f1b-2tmbn\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:56.931270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:56.931205    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:42:57.095972 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:57.095932    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs" podUID="27120312-1468-480d-9899-8768fd4593d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:57.252668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:57.252600    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"]
Oct 30 22:42:57.254326 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:42:57.254282    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fff5de9_809a_446c_aa19_79020a584b4a.slice/crio-4cdf1a39613165a00a5bb276c778fea1e425ccba227330f0f6543601964592d2 WatchSource:0}: Error finding container 4cdf1a39613165a00a5bb276c778fea1e425ccba227330f0f6543601964592d2: Status 404 returned error can't find the container with id 4cdf1a39613165a00a5bb276c778fea1e425ccba227330f0f6543601964592d2
Oct 30 22:42:57.882080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:57.882041    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn" event={"ID":"7fff5de9-809a-446c-aa19-79020a584b4a","Type":"ContainerStarted","Data":"bd37296461c3cca1a64b6ef0ee2e31c669444374ff8a406aa7ac9dc72840c805"}
Oct 30 22:42:57.882557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:57.882114    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn" event={"ID":"7fff5de9-809a-446c-aa19-79020a584b4a","Type":"ContainerStarted","Data":"4cdf1a39613165a00a5bb276c778fea1e425ccba227330f0f6543601964592d2"}
Oct 30 22:42:57.917696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:57.917638    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn" podStartSLOduration=1.917620661 podStartE2EDuration="1.917620661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:57.91597148 +0000 UTC m=+2161.954910366" watchObservedRunningTime="2026-10-30 22:42:57.917620661 +0000 UTC m=+2161.956559547"
Oct 30 22:42:58.893754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:58.893715    4581 generic.go:356] "Generic (PLEG): container finished" podID="7fff5de9-809a-446c-aa19-79020a584b4a" containerID="bd37296461c3cca1a64b6ef0ee2e31c669444374ff8a406aa7ac9dc72840c805" exitCode=0
Oct 30 22:42:58.894321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:42:58.893812    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn" event={"ID":"7fff5de9-809a-446c-aa19-79020a584b4a","Type":"ContainerDied","Data":"bd37296461c3cca1a64b6ef0ee2e31c669444374ff8a406aa7ac9dc72840c805"}
Oct 30 22:43:00.090112 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.090080    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:43:00.226724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.226689    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fff5de9-809a-446c-aa19-79020a584b4a-kube-api-access-r7mxb\" (UniqueName: \"kubernetes.io/projected/7fff5de9-809a-446c-aa19-79020a584b4a-kube-api-access-r7mxb\") pod \"7fff5de9-809a-446c-aa19-79020a584b4a\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") "
Oct 30 22:43:00.226915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.226857    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-xks-ca\") pod \"7fff5de9-809a-446c-aa19-79020a584b4a\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") "
Oct 30 22:43:00.226989 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.226940    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-openshift-ca\") pod \"7fff5de9-809a-446c-aa19-79020a584b4a\" (UID: \"7fff5de9-809a-446c-aa19-79020a584b4a\") "
Oct 30 22:43:00.227328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.227294    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-openshift-ca" pod "7fff5de9-809a-446c-aa19-79020a584b4a" (UID: "7fff5de9-809a-446c-aa19-79020a584b4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:00.227738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.227717    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:00.229107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.229084    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fff5de9-809a-446c-aa19-79020a584b4a-kube-api-access-r7mxb" pod "7fff5de9-809a-446c-aa19-79020a584b4a" (UID: "7fff5de9-809a-446c-aa19-79020a584b4a"). InnerVolumeSpecName "kube-api-access-r7mxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:00.229182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.229105    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-xks-ca" pod "7fff5de9-809a-446c-aa19-79020a584b4a" (UID: "7fff5de9-809a-446c-aa19-79020a584b4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:00.329204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.329146    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7mxb\" (UniqueName: \"kubernetes.io/projected/7fff5de9-809a-446c-aa19-79020a584b4a-kube-api-access-r7mxb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:00.329204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.329198    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fff5de9-809a-446c-aa19-79020a584b4a-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:00.913646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.913615    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"
Oct 30 22:43:00.913850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.913651    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn" event={"ID":"7fff5de9-809a-446c-aa19-79020a584b4a","Type":"ContainerDied","Data":"4cdf1a39613165a00a5bb276c778fea1e425ccba227330f0f6543601964592d2"}
Oct 30 22:43:00.913850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:00.913702    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cdf1a39613165a00a5bb276c778fea1e425ccba227330f0f6543601964592d2"
Oct 30 22:43:07.097591 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:07.097551    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs" podUID="27120312-1468-480d-9899-8768fd4593d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:43:10.821147 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:43:10.821106    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:43:12.828367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:12.828336    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:43:12.926782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:12.926745    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27120312-1468-480d-9899-8768fd4593d3-kube-api-access-hrs6z\" (UniqueName: \"kubernetes.io/projected/27120312-1468-480d-9899-8768fd4593d3-kube-api-access-hrs6z\") pod \"27120312-1468-480d-9899-8768fd4593d3\" (UID: \"27120312-1468-480d-9899-8768fd4593d3\") "
Oct 30 22:43:12.927001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:12.926902    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27120312-1468-480d-9899-8768fd4593d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27120312-1468-480d-9899-8768fd4593d3-plugins-config-volume\") pod \"27120312-1468-480d-9899-8768fd4593d3\" (UID: \"27120312-1468-480d-9899-8768fd4593d3\") "
Oct 30 22:43:12.927481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:12.927451    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27120312-1468-480d-9899-8768fd4593d3-plugins-config-volume" pod "27120312-1468-480d-9899-8768fd4593d3" (UID: "27120312-1468-480d-9899-8768fd4593d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:12.927941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:12.927920    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27120312-1468-480d-9899-8768fd4593d3-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:12.929712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:12.929680    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27120312-1468-480d-9899-8768fd4593d3-kube-api-access-hrs6z" pod "27120312-1468-480d-9899-8768fd4593d3" (UID: "27120312-1468-480d-9899-8768fd4593d3"). InnerVolumeSpecName "kube-api-access-hrs6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:13.029521 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.029487    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrs6z\" (UniqueName: \"kubernetes.io/projected/27120312-1468-480d-9899-8768fd4593d3-kube-api-access-hrs6z\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:13.051107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.051003    4581 generic.go:356] "Generic (PLEG): container finished" podID="27120312-1468-480d-9899-8768fd4593d3" containerID="2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e" exitCode=0
Oct 30 22:43:13.051107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.051049    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"
Oct 30 22:43:13.051397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.051092    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs" event={"ID":"27120312-1468-480d-9899-8768fd4593d3","Type":"ContainerDied","Data":"2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e"}
Oct 30 22:43:13.051397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.051160    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs" event={"ID":"27120312-1468-480d-9899-8768fd4593d3","Type":"ContainerDied","Data":"334219a7c247959a08b502eb12bb63bd06025aa095b3c573ed719587a529c130"}
Oct 30 22:43:13.051397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.051193    4581 scope.go:122] "RemoveContainer" containerID="2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e"
Oct 30 22:43:13.065513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.065485    4581 scope.go:122] "RemoveContainer" containerID="2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e"
Oct 30 22:43:13.065816 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:43:13.065792    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e\": container with ID starting with 2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e not found: ID does not exist" containerID="2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e"
Oct 30 22:43:13.065893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.065825    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e"} err="failed to get container status \"2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e\": rpc error: code = NotFound desc = could not find container \"2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e\": container with ID starting with 2463cd17cedaea870a3e14f5a73ed5c4e0d11ba67bf56052fb06949b28cbfb4e not found: ID does not exist"
Oct 30 22:43:13.102321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.102285    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"]
Oct 30 22:43:13.107938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:13.107903    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9d87b-d46894c99-vcmjs"]
Oct 30 22:43:14.102735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:14.102702    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27120312-1468-480d-9899-8768fd4593d3" path="/var/lib/kubelet/pods/27120312-1468-480d-9899-8768fd4593d3/volumes"
Oct 30 22:43:16.469660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.469629    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"]
Oct 30 22:43:16.470431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.470416    4581 cpu_manager.go:414] "containerMap: removing container" podUID="27120312-1468-480d-9899-8768fd4593d3" containerName="payload-pre-processing"
Oct 30 22:43:16.470493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.470433    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27120312-1468-480d-9899-8768fd4593d3" containerName="payload-pre-processing"
Oct 30 22:43:16.470493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.470451    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7fff5de9-809a-446c-aa19-79020a584b4a" containerName="revoke-keys"
Oct 30 22:43:16.470493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.470456    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fff5de9-809a-446c-aa19-79020a584b4a" containerName="revoke-keys"
Oct 30 22:43:16.470618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.470608    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="27120312-1468-480d-9899-8768fd4593d3" containerName="payload-pre-processing"
Oct 30 22:43:16.470654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.470621    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fff5de9-809a-446c-aa19-79020a584b4a" containerName="revoke-keys"
Oct 30 22:43:16.476903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.476881    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.500657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.500629    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:16.500657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.500640    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:16.500657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.500665    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:16.500903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.500703    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:43:16.507002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.506977    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"]
Oct 30 22:43:16.592895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.592847    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.593086 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.592900    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.593336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.593306    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthfv\" (UniqueName: \"kubernetes.io/projected/77841501-45f6-4da3-9b7a-150e819f1a23-kube-api-access-pthfv\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.694968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.694929    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthfv\" (UniqueName: \"kubernetes.io/projected/77841501-45f6-4da3-9b7a-150e819f1a23-kube-api-access-pthfv\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.695152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.695079    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.695152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.695106    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.695865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.695835    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.697804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.697782    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.712854 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.712829    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthfv\" (UniqueName: \"kubernetes.io/projected/77841501-45f6-4da3-9b7a-150e819f1a23-kube-api-access-pthfv\") pod \"maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:16.793465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:16.793379    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:17.114907 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:17.114881    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"]
Oct 30 22:43:17.117742 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:43:17.117710    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77841501_45f6_4da3_9b7a_150e819f1a23.slice/crio-1f0527e66b24372070c05c04c50aaddaff8ea195a4d28e79312349778f3eaa86 WatchSource:0}: Error finding container 1f0527e66b24372070c05c04c50aaddaff8ea195a4d28e79312349778f3eaa86: Status 404 returned error can't find the container with id 1f0527e66b24372070c05c04c50aaddaff8ea195a4d28e79312349778f3eaa86
Oct 30 22:43:18.107290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:18.107250    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f" event={"ID":"77841501-45f6-4da3-9b7a-150e819f1a23","Type":"ContainerStarted","Data":"8d31ac090cf826ef6985071d4148a59ebba8b3e72cba5f0c286d7714da9f9a6a"}
Oct 30 22:43:18.107727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:18.107319    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f" event={"ID":"77841501-45f6-4da3-9b7a-150e819f1a23","Type":"ContainerStarted","Data":"1f0527e66b24372070c05c04c50aaddaff8ea195a4d28e79312349778f3eaa86"}
Oct 30 22:43:18.147979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:18.147918    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f" podStartSLOduration=2.147900794 podStartE2EDuration="2.147900794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:18.141414006 +0000 UTC m=+2182.180352892" watchObservedRunningTime="2026-10-30 22:43:18.147900794 +0000 UTC m=+2182.186839679"
Oct 30 22:43:19.117881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:19.117844    4581 generic.go:356] "Generic (PLEG): container finished" podID="77841501-45f6-4da3-9b7a-150e819f1a23" containerID="8d31ac090cf826ef6985071d4148a59ebba8b3e72cba5f0c286d7714da9f9a6a" exitCode=0
Oct 30 22:43:19.118374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:19.117925    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f" event={"ID":"77841501-45f6-4da3-9b7a-150e819f1a23","Type":"ContainerDied","Data":"8d31ac090cf826ef6985071d4148a59ebba8b3e72cba5f0c286d7714da9f9a6a"}
Oct 30 22:43:20.505280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.476695    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:20.579444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.566903    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77841501-45f6-4da3-9b7a-150e819f1a23-kube-api-access-pthfv\" (UniqueName: \"kubernetes.io/projected/77841501-45f6-4da3-9b7a-150e819f1a23-kube-api-access-pthfv\") pod \"77841501-45f6-4da3-9b7a-150e819f1a23\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") "
Oct 30 22:43:20.612084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.605331    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-xks-ca\") pod \"77841501-45f6-4da3-9b7a-150e819f1a23\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") "
Oct 30 22:43:20.612084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.605456    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-openshift-ca\") pod \"77841501-45f6-4da3-9b7a-150e819f1a23\" (UID: \"77841501-45f6-4da3-9b7a-150e819f1a23\") "
Oct 30 22:43:20.615732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.570878    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77841501-45f6-4da3-9b7a-150e819f1a23-kube-api-access-pthfv" pod "77841501-45f6-4da3-9b7a-150e819f1a23" (UID: "77841501-45f6-4da3-9b7a-150e819f1a23"). InnerVolumeSpecName "kube-api-access-pthfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:20.620280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.620194    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-openshift-ca" pod "77841501-45f6-4da3-9b7a-150e819f1a23" (UID: "77841501-45f6-4da3-9b7a-150e819f1a23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:20.627978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.625254    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthfv\" (UniqueName: \"kubernetes.io/projected/77841501-45f6-4da3-9b7a-150e819f1a23-kube-api-access-pthfv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.627978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.625288    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:20.647455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.647418    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-xks-ca" pod "77841501-45f6-4da3-9b7a-150e819f1a23" (UID: "77841501-45f6-4da3-9b7a-150e819f1a23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:20.729140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:20.729092    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77841501-45f6-4da3-9b7a-150e819f1a23-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:21.138572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:21.138547    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"
Oct 30 22:43:21.138760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:21.138589    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f" event={"ID":"77841501-45f6-4da3-9b7a-150e819f1a23","Type":"ContainerDied","Data":"1f0527e66b24372070c05c04c50aaddaff8ea195a4d28e79312349778f3eaa86"}
Oct 30 22:43:21.138760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:21.138634    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f0527e66b24372070c05c04c50aaddaff8ea195a4d28e79312349778f3eaa86"
Oct 30 22:43:26.920941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:26.920905    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"]
Oct 30 22:43:26.956359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:26.956329    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"]
Oct 30 22:43:31.921847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:31.921801    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q" podUID="520d7bde-53bf-42a2-a31a-17555e1981a8" containerName="payload-processing" containerID="cri-o://de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a" gracePeriod=25
Oct 30 22:43:31.956892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:31.956816    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl" podUID="841bf7d5-6a07-4d7c-a170-bd39df91d349" containerName="payload-pre-processing" containerID="cri-o://1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d" gracePeriod=25
Oct 30 22:43:32.246141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.246117    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:43:32.369570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.369538    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520d7bde-53bf-42a2-a31a-17555e1981a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520d7bde-53bf-42a2-a31a-17555e1981a8-plugins-config-volume\") pod \"520d7bde-53bf-42a2-a31a-17555e1981a8\" (UID: \"520d7bde-53bf-42a2-a31a-17555e1981a8\") "
Oct 30 22:43:32.369780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.369606    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520d7bde-53bf-42a2-a31a-17555e1981a8-kube-api-access-m42s7\" (UniqueName: \"kubernetes.io/projected/520d7bde-53bf-42a2-a31a-17555e1981a8-kube-api-access-m42s7\") pod \"520d7bde-53bf-42a2-a31a-17555e1981a8\" (UID: \"520d7bde-53bf-42a2-a31a-17555e1981a8\") "
Oct 30 22:43:32.370002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.369974    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520d7bde-53bf-42a2-a31a-17555e1981a8-plugins-config-volume" pod "520d7bde-53bf-42a2-a31a-17555e1981a8" (UID: "520d7bde-53bf-42a2-a31a-17555e1981a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:32.370445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.370426    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520d7bde-53bf-42a2-a31a-17555e1981a8-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.372473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.372449    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520d7bde-53bf-42a2-a31a-17555e1981a8-kube-api-access-m42s7" pod "520d7bde-53bf-42a2-a31a-17555e1981a8" (UID: "520d7bde-53bf-42a2-a31a-17555e1981a8"). InnerVolumeSpecName "kube-api-access-m42s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:32.383357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.383328    4581 generic.go:356] "Generic (PLEG): container finished" podID="520d7bde-53bf-42a2-a31a-17555e1981a8" containerID="de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a" exitCode=0
Oct 30 22:43:32.383523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.383385    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"
Oct 30 22:43:32.383523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.383431    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q" event={"ID":"520d7bde-53bf-42a2-a31a-17555e1981a8","Type":"ContainerDied","Data":"de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a"}
Oct 30 22:43:32.383523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.383505    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q" event={"ID":"520d7bde-53bf-42a2-a31a-17555e1981a8","Type":"ContainerDied","Data":"ec95b09eae8ef9df5ed2ec859261390dd2d06268ee69511742e47bab22fd3766"}
Oct 30 22:43:32.383760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.383538    4581 scope.go:122] "RemoveContainer" containerID="de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a"
Oct 30 22:43:32.400007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.399990    4581 scope.go:122] "RemoveContainer" containerID="de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a"
Oct 30 22:43:32.400345 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:43:32.400326    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a\": container with ID starting with de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a not found: ID does not exist" containerID="de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a"
Oct 30 22:43:32.400415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.400353    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a"} err="failed to get container status \"de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a\": rpc error: code = NotFound desc = could not find container \"de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a\": container with ID starting with de28c01baaf3e73ea7798bcd1cbdad14ad4e0ecdfe8a84250712dc832670be8a not found: ID does not exist"
Oct 30 22:43:32.437232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.437194    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"]
Oct 30 22:43:32.442478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.442450    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdd94b-5f4f4579c9-4mq7q"]
Oct 30 22:43:32.471860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:32.471796    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42s7\" (UniqueName: \"kubernetes.io/projected/520d7bde-53bf-42a2-a31a-17555e1981a8-kube-api-access-m42s7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:34.099269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:34.099189    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520d7bde-53bf-42a2-a31a-17555e1981a8" path="/var/lib/kubelet/pods/520d7bde-53bf-42a2-a31a-17555e1981a8/volumes"
Oct 30 22:43:35.388388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:35.388351    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"]
Oct 30 22:43:35.389402 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:35.389373    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f9d87b-jmznd"]
Oct 30 22:43:36.099250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:36.099187    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b1a190-38f2-4bf5-a66b-5c13bb5cee41" path="/var/lib/kubelet/pods/52b1a190-38f2-4bf5-a66b-5c13bb5cee41/volumes"
Oct 30 22:43:37.463505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:37.463464    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl" podUID="841bf7d5-6a07-4d7c-a170-bd39df91d349" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:43:40.818415 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:43:40.818380    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:43:43.013652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:43.012409    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"]
Oct 30 22:43:43.013652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:43.012659    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy" containerID="cri-o://190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8" gracePeriod=30
Oct 30 22:43:47.036867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:47.036829    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"]
Oct 30 22:43:47.042417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:47.042389    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdd94b-4jb2f"]
Oct 30 22:43:47.462013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:47.461975    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl" podUID="841bf7d5-6a07-4d7c-a170-bd39df91d349" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:43:48.101138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.101097    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77841501-45f6-4da3-9b7a-150e819f1a23" path="/var/lib/kubelet/pods/77841501-45f6-4da3-9b7a-150e819f1a23/volumes"
Oct 30 22:43:48.330874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.330846    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:43:48.512879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.512843    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-certs\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.512983    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-data\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513010    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-token\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-token\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513142    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-podinfo\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513153    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-certs" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.513285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513212    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-socket\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513279    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-data" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.513341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513308    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d006e94-53e2-4e63-9b5e-1632efdc3194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d006e94-53e2-4e63-9b5e-1632efdc3194-istiod-ca-cert\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513355    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-credential-socket\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513415    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-kube-api-access-82rkq\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-kube-api-access-82rkq\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513496    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-envoy\") pod \"5d006e94-53e2-4e63-9b5e-1632efdc3194\" (UID: \"5d006e94-53e2-4e63-9b5e-1632efdc3194\") "
Oct 30 22:43:48.513542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513490    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-socket" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.513662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513644    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-credential-socket" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.513717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.513663    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d006e94-53e2-4e63-9b5e-1632efdc3194-istiod-ca-cert" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:48.514518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.514490    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.514708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.514520    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d006e94-53e2-4e63-9b5e-1632efdc3194-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.514708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.514537    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.514708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.514553    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.514708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.514569    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.515630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.515607    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-podinfo" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:43:48.515750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.515730    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-token" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:48.515810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.515771    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-envoy" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:48.515849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.515826    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-kube-api-access-82rkq" pod "5d006e94-53e2-4e63-9b5e-1632efdc3194" (UID: "5d006e94-53e2-4e63-9b5e-1632efdc3194"). InnerVolumeSpecName "kube-api-access-82rkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:48.565265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.565139    4581 generic.go:356] "Generic (PLEG): container finished" podID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerID="190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8" exitCode=0
Oct 30 22:43:48.565513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.565210    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"
Oct 30 22:43:48.568911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.565248    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" event={"ID":"5d006e94-53e2-4e63-9b5e-1632efdc3194","Type":"ContainerDied","Data":"190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8"}
Oct 30 22:43:48.568911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.565736    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d" event={"ID":"5d006e94-53e2-4e63-9b5e-1632efdc3194","Type":"ContainerDied","Data":"b3d362fcc7921a33e100bc33259eaa07e3bd861f828fdbaeba3c5302da3ecf3b"}
Oct 30 22:43:48.568911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.565775    4581 scope.go:122] "RemoveContainer" containerID="190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8"
Oct 30 22:43:48.597780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.597748    4581 scope.go:122] "RemoveContainer" containerID="190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8"
Oct 30 22:43:48.598195 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:43:48.598169    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8\": container with ID starting with 190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8 not found: ID does not exist" containerID="190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8"
Oct 30 22:43:48.598369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.598206    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8"} err="failed to get container status \"190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8\": rpc error: code = NotFound desc = could not find container \"190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8\": container with ID starting with 190ba9fd8103be7d6cb832b86e58d933aa9ba3d7a0794aab909da4a1e4e8f2f8 not found: ID does not exist"
Oct 30 22:43:48.616096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.616069    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.616096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.616093    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.616288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.616103    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rkq\" (UniqueName: \"kubernetes.io/projected/5d006e94-53e2-4e63-9b5e-1632efdc3194-kube-api-access-82rkq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.616288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.616112    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d006e94-53e2-4e63-9b5e-1632efdc3194-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:48.623457 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.623430    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"]
Oct 30 22:43:48.628397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:48.628376    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f9d87b-openshift-default-6fb7566b69-5hv5d"]
Oct 30 22:43:50.099796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:50.099762    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" path="/var/lib/kubelet/pods/5d006e94-53e2-4e63-9b5e-1632efdc3194/volumes"
Oct 30 22:43:52.275900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.275869    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:43:52.376268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.376232    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841bf7d5-6a07-4d7c-a170-bd39df91d349-kube-api-access-k86dw\" (UniqueName: \"kubernetes.io/projected/841bf7d5-6a07-4d7c-a170-bd39df91d349-kube-api-access-k86dw\") pod \"841bf7d5-6a07-4d7c-a170-bd39df91d349\" (UID: \"841bf7d5-6a07-4d7c-a170-bd39df91d349\") "
Oct 30 22:43:52.376474 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.376375    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841bf7d5-6a07-4d7c-a170-bd39df91d349-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841bf7d5-6a07-4d7c-a170-bd39df91d349-plugins-config-volume\") pod \"841bf7d5-6a07-4d7c-a170-bd39df91d349\" (UID: \"841bf7d5-6a07-4d7c-a170-bd39df91d349\") "
Oct 30 22:43:52.376906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.376826    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841bf7d5-6a07-4d7c-a170-bd39df91d349-plugins-config-volume" pod "841bf7d5-6a07-4d7c-a170-bd39df91d349" (UID: "841bf7d5-6a07-4d7c-a170-bd39df91d349"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:52.377130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.377109    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841bf7d5-6a07-4d7c-a170-bd39df91d349-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:52.379449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.379412    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841bf7d5-6a07-4d7c-a170-bd39df91d349-kube-api-access-k86dw" pod "841bf7d5-6a07-4d7c-a170-bd39df91d349" (UID: "841bf7d5-6a07-4d7c-a170-bd39df91d349"). InnerVolumeSpecName "kube-api-access-k86dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:52.478543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.478509    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86dw\" (UniqueName: \"kubernetes.io/projected/841bf7d5-6a07-4d7c-a170-bd39df91d349-kube-api-access-k86dw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:52.621378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.621344    4581 generic.go:356] "Generic (PLEG): container finished" podID="841bf7d5-6a07-4d7c-a170-bd39df91d349" containerID="1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d" exitCode=0
Oct 30 22:43:52.621571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.621390    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"
Oct 30 22:43:52.621571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.621389    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl" event={"ID":"841bf7d5-6a07-4d7c-a170-bd39df91d349","Type":"ContainerDied","Data":"1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d"}
Oct 30 22:43:52.621571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.621444    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl" event={"ID":"841bf7d5-6a07-4d7c-a170-bd39df91d349","Type":"ContainerDied","Data":"5620e9bb47afd8f19132d61740f1653f9b6f6031c1bf30b02f9f447e5356ef17"}
Oct 30 22:43:52.621571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.621466    4581 scope.go:122] "RemoveContainer" containerID="1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d"
Oct 30 22:43:52.634116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.634098    4581 scope.go:122] "RemoveContainer" containerID="1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d"
Oct 30 22:43:52.634416 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:43:52.634393    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d\": container with ID starting with 1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d not found: ID does not exist" containerID="1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d"
Oct 30 22:43:52.634485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.634424    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d"} err="failed to get container status \"1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d\": rpc error: code = NotFound desc = could not find container \"1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d\": container with ID starting with 1f8c5645cd2907c608817fe98638e94394b6cc7fee752ceed1056aa68b0d300d not found: ID does not exist"
Oct 30 22:43:52.669083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.669050    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"]
Oct 30 22:43:52.677720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:52.677688    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdd94b-74d9b684bc-m4kfl"]
Oct 30 22:43:54.100514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:54.100480    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841bf7d5-6a07-4d7c-a170-bd39df91d349" path="/var/lib/kubelet/pods/841bf7d5-6a07-4d7c-a170-bd39df91d349/volumes"
Oct 30 22:43:55.742692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.742622    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"]
Oct 30 22:43:55.743825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.743789    4581 cpu_manager.go:414] "containerMap: removing container" podUID="841bf7d5-6a07-4d7c-a170-bd39df91d349" containerName="payload-pre-processing"
Oct 30 22:43:55.743825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.743814    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841bf7d5-6a07-4d7c-a170-bd39df91d349" containerName="payload-pre-processing"
Oct 30 22:43:55.743998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.743866    4581 cpu_manager.go:414] "containerMap: removing container" podUID="520d7bde-53bf-42a2-a31a-17555e1981a8" containerName="payload-processing"
Oct 30 22:43:55.743998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.743873    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520d7bde-53bf-42a2-a31a-17555e1981a8" containerName="payload-processing"
Oct 30 22:43:55.743998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.743888    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy"
Oct 30 22:43:55.743998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.743895    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy"
Oct 30 22:43:55.743998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.743930    4581 cpu_manager.go:414] "containerMap: removing container" podUID="77841501-45f6-4da3-9b7a-150e819f1a23" containerName="revoke-keys"
Oct 30 22:43:55.743998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.743937    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77841501-45f6-4da3-9b7a-150e819f1a23" containerName="revoke-keys"
Oct 30 22:43:55.744369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.744125    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="841bf7d5-6a07-4d7c-a170-bd39df91d349" containerName="payload-pre-processing"
Oct 30 22:43:55.744369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.744152    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="520d7bde-53bf-42a2-a31a-17555e1981a8" containerName="payload-processing"
Oct 30 22:43:55.744369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.744163    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d006e94-53e2-4e63-9b5e-1632efdc3194" containerName="istio-proxy"
Oct 30 22:43:55.744369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.744177    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="77841501-45f6-4da3-9b7a-150e819f1a23" containerName="revoke-keys"
Oct 30 22:43:55.750486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.750464    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.757054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.757027    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.757296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.757275    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.757387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.757341    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7qj\" (UniqueName: \"kubernetes.io/projected/1397c723-9c26-42fa-b712-51f4aa20abc5-kube-api-access-dp7qj\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.764032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.764013    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:55.772855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.772831    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:55.772855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.772833    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:55.773137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.773114    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:43:55.778720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.778697    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"]
Oct 30 22:43:55.858755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.858720    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.858971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.858836    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.858971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.858879    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7qj\" (UniqueName: \"kubernetes.io/projected/1397c723-9c26-42fa-b712-51f4aa20abc5-kube-api-access-dp7qj\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.859552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.859530    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.861447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.861425    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:55.878543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:55.878519    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7qj\" (UniqueName: \"kubernetes.io/projected/1397c723-9c26-42fa-b712-51f4aa20abc5-kube-api-access-dp7qj\") pod \"maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:56.069257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:56.069144    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:56.388931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:56.388907    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"]
Oct 30 22:43:56.390866 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:43:56.390834    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1397c723_9c26_42fa_b712_51f4aa20abc5.slice/crio-d0e2040b45528d80f3eca27d569933473a9373d9ef7cccbb2cf58f9a1a5d4406 WatchSource:0}: Error finding container d0e2040b45528d80f3eca27d569933473a9373d9ef7cccbb2cf58f9a1a5d4406: Status 404 returned error can't find the container with id d0e2040b45528d80f3eca27d569933473a9373d9ef7cccbb2cf58f9a1a5d4406
Oct 30 22:43:56.719672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:56.719583    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg" event={"ID":"1397c723-9c26-42fa-b712-51f4aa20abc5","Type":"ContainerStarted","Data":"d0e2040b45528d80f3eca27d569933473a9373d9ef7cccbb2cf58f9a1a5d4406"}
Oct 30 22:43:57.746037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:57.745942    4581 generic.go:356] "Generic (PLEG): container finished" podID="1397c723-9c26-42fa-b712-51f4aa20abc5" containerID="7110d24ca1c4e1f3d62d4852aefaf743c32e034a72b63bb8c39a711b1f140cf1" exitCode=0
Oct 30 22:43:57.746679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:57.746009    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg" event={"ID":"1397c723-9c26-42fa-b712-51f4aa20abc5","Type":"ContainerDied","Data":"7110d24ca1c4e1f3d62d4852aefaf743c32e034a72b63bb8c39a711b1f140cf1"}
Oct 30 22:43:58.976332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:58.976307    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:59.021717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.021684    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-openshift-ca\") pod \"1397c723-9c26-42fa-b712-51f4aa20abc5\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") "
Oct 30 22:43:59.021906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.021819    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-xks-ca\") pod \"1397c723-9c26-42fa-b712-51f4aa20abc5\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") "
Oct 30 22:43:59.021985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.021941    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1397c723-9c26-42fa-b712-51f4aa20abc5-kube-api-access-dp7qj\" (UniqueName: \"kubernetes.io/projected/1397c723-9c26-42fa-b712-51f4aa20abc5-kube-api-access-dp7qj\") pod \"1397c723-9c26-42fa-b712-51f4aa20abc5\" (UID: \"1397c723-9c26-42fa-b712-51f4aa20abc5\") "
Oct 30 22:43:59.022078 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.022036    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-openshift-ca" pod "1397c723-9c26-42fa-b712-51f4aa20abc5" (UID: "1397c723-9c26-42fa-b712-51f4aa20abc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:59.022754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.022722    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:59.024179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.024149    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-xks-ca" pod "1397c723-9c26-42fa-b712-51f4aa20abc5" (UID: "1397c723-9c26-42fa-b712-51f4aa20abc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:59.024373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.024275    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1397c723-9c26-42fa-b712-51f4aa20abc5-kube-api-access-dp7qj" pod "1397c723-9c26-42fa-b712-51f4aa20abc5" (UID: "1397c723-9c26-42fa-b712-51f4aa20abc5"). InnerVolumeSpecName "kube-api-access-dp7qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:59.124518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.124478    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp7qj\" (UniqueName: \"kubernetes.io/projected/1397c723-9c26-42fa-b712-51f4aa20abc5-kube-api-access-dp7qj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:59.124518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.124507    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1397c723-9c26-42fa-b712-51f4aa20abc5-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:43:59.793426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.793387    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"
Oct 30 22:43:59.793426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.793416    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg" event={"ID":"1397c723-9c26-42fa-b712-51f4aa20abc5","Type":"ContainerDied","Data":"d0e2040b45528d80f3eca27d569933473a9373d9ef7cccbb2cf58f9a1a5d4406"}
Oct 30 22:43:59.793710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:43:59.793465    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0e2040b45528d80f3eca27d569933473a9373d9ef7cccbb2cf58f9a1a5d4406"
Oct 30 22:44:06.181791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:06.181747    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"]
Oct 30 22:44:06.222062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:06.222027    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"]
Oct 30 22:44:10.818784 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:44:10.818744    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:44:11.182093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.182048    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg" podUID="3f20acdb-7d98-459b-b668-91f1d4784470" containerName="payload-processing" containerID="cri-o://4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445" gracePeriod=25
Oct 30 22:44:11.222903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.222858    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw" podUID="0899aba0-a556-4a34-a0db-d6205a977d62" containerName="payload-pre-processing" containerID="cri-o://4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f" gracePeriod=25
Oct 30 22:44:11.555566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.555535    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:44:11.582328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.582298    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f20acdb-7d98-459b-b668-91f1d4784470-kube-api-access-sbrsh\" (UniqueName: \"kubernetes.io/projected/3f20acdb-7d98-459b-b668-91f1d4784470-kube-api-access-sbrsh\") pod \"3f20acdb-7d98-459b-b668-91f1d4784470\" (UID: \"3f20acdb-7d98-459b-b668-91f1d4784470\") "
Oct 30 22:44:11.584840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.584809    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f20acdb-7d98-459b-b668-91f1d4784470-kube-api-access-sbrsh" pod "3f20acdb-7d98-459b-b668-91f1d4784470" (UID: "3f20acdb-7d98-459b-b668-91f1d4784470"). InnerVolumeSpecName "kube-api-access-sbrsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:11.684064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.684028    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f20acdb-7d98-459b-b668-91f1d4784470-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f20acdb-7d98-459b-b668-91f1d4784470-plugins-config-volume\") pod \"3f20acdb-7d98-459b-b668-91f1d4784470\" (UID: \"3f20acdb-7d98-459b-b668-91f1d4784470\") "
Oct 30 22:44:11.684456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.684433    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f20acdb-7d98-459b-b668-91f1d4784470-plugins-config-volume" pod "3f20acdb-7d98-459b-b668-91f1d4784470" (UID: "3f20acdb-7d98-459b-b668-91f1d4784470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:11.684664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.684649    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f20acdb-7d98-459b-b668-91f1d4784470-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:44:11.684709 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:11.684666    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbrsh\" (UniqueName: \"kubernetes.io/projected/3f20acdb-7d98-459b-b668-91f1d4784470-kube-api-access-sbrsh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:44:12.001106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.001070    4581 generic.go:356] "Generic (PLEG): container finished" podID="3f20acdb-7d98-459b-b668-91f1d4784470" containerID="4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445" exitCode=0
Oct 30 22:44:12.001607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.001118    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"
Oct 30 22:44:12.001607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.001119    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg" event={"ID":"3f20acdb-7d98-459b-b668-91f1d4784470","Type":"ContainerDied","Data":"4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445"}
Oct 30 22:44:12.001607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.001276    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg" event={"ID":"3f20acdb-7d98-459b-b668-91f1d4784470","Type":"ContainerDied","Data":"58fa39368fa8b23acee77505e76502a7b904ba62f1c5b0b1ed248e2909d76ff4"}
Oct 30 22:44:12.001607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.001313    4581 scope.go:122] "RemoveContainer" containerID="4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445"
Oct 30 22:44:12.013637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.013615    4581 scope.go:122] "RemoveContainer" containerID="4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445"
Oct 30 22:44:12.013919 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:44:12.013900    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445\": container with ID starting with 4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445 not found: ID does not exist" containerID="4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445"
Oct 30 22:44:12.013968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.013925    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445"} err="failed to get container status \"4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445\": rpc error: code = NotFound desc = could not find container \"4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445\": container with ID starting with 4523895bfe1fe621f66384d9b56e49adec7837be278234d738504e29b03a4445 not found: ID does not exist"
Oct 30 22:44:12.049013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.048991    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"]
Oct 30 22:44:12.060969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.060941    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6013c8-7678cc4db9-mppqg"]
Oct 30 22:44:12.101590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:12.101553    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f20acdb-7d98-459b-b668-91f1d4784470" path="/var/lib/kubelet/pods/3f20acdb-7d98-459b-b668-91f1d4784470/volumes"
Oct 30 22:44:21.061864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:21.061826    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw" podUID="0899aba0-a556-4a34-a0db-d6205a977d62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:44:31.062077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:31.062036    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw" podUID="0899aba0-a556-4a34-a0db-d6205a977d62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:44:31.548150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:31.548118    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:44:31.606919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:31.606892    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0899aba0-a556-4a34-a0db-d6205a977d62-kube-api-access-q5czb\" (UniqueName: \"kubernetes.io/projected/0899aba0-a556-4a34-a0db-d6205a977d62-kube-api-access-q5czb\") pod \"0899aba0-a556-4a34-a0db-d6205a977d62\" (UID: \"0899aba0-a556-4a34-a0db-d6205a977d62\") "
Oct 30 22:44:31.607110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:31.607071    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0899aba0-a556-4a34-a0db-d6205a977d62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899aba0-a556-4a34-a0db-d6205a977d62-plugins-config-volume\") pod \"0899aba0-a556-4a34-a0db-d6205a977d62\" (UID: \"0899aba0-a556-4a34-a0db-d6205a977d62\") "
Oct 30 22:44:31.607861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:31.607831    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0899aba0-a556-4a34-a0db-d6205a977d62-plugins-config-volume" pod "0899aba0-a556-4a34-a0db-d6205a977d62" (UID: "0899aba0-a556-4a34-a0db-d6205a977d62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:31.630647 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:31.630617    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0899aba0-a556-4a34-a0db-d6205a977d62-kube-api-access-q5czb" pod "0899aba0-a556-4a34-a0db-d6205a977d62" (UID: "0899aba0-a556-4a34-a0db-d6205a977d62"). InnerVolumeSpecName "kube-api-access-q5czb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:31.710915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:31.709544    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899aba0-a556-4a34-a0db-d6205a977d62-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.720109 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:31.711978    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5czb\" (UniqueName: \"kubernetes.io/projected/0899aba0-a556-4a34-a0db-d6205a977d62-kube-api-access-q5czb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:44:32.285722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.285690    4581 generic.go:356] "Generic (PLEG): container finished" podID="0899aba0-a556-4a34-a0db-d6205a977d62" containerID="4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f" exitCode=0
Oct 30 22:44:32.286209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.285744    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"
Oct 30 22:44:32.286209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.285744    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw" event={"ID":"0899aba0-a556-4a34-a0db-d6205a977d62","Type":"ContainerDied","Data":"4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f"}
Oct 30 22:44:32.286209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.285883    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw" event={"ID":"0899aba0-a556-4a34-a0db-d6205a977d62","Type":"ContainerDied","Data":"b6d2bb241246f66df8fa4ff6da39e142ea80eaf1483c5fc8501ea43a64504a05"}
Oct 30 22:44:32.286209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.285915    4581 scope.go:122] "RemoveContainer" containerID="4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f"
Oct 30 22:44:32.298469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.298442    4581 scope.go:122] "RemoveContainer" containerID="4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f"
Oct 30 22:44:32.298818 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:44:32.298789    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f\": container with ID starting with 4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f not found: ID does not exist" containerID="4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f"
Oct 30 22:44:32.298925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.298827    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f"} err="failed to get container status \"4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f\": rpc error: code = NotFound desc = could not find container \"4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f\": container with ID starting with 4d791a2f2fb963931a1fc3b927fa1db9aa268a6e59b2a9f2f0081c6a0ff2af6f not found: ID does not exist"
Oct 30 22:44:32.333774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.333745    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"]
Oct 30 22:44:32.343278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:32.343244    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6013c8-94b84fcd7-b54fw"]
Oct 30 22:44:34.100804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:34.100770    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0899aba0-a556-4a34-a0db-d6205a977d62" path="/var/lib/kubelet/pods/0899aba0-a556-4a34-a0db-d6205a977d62/volumes"
Oct 30 22:44:40.819511 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:44:40.819469    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:44:47.983093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:47.983060    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"]
Oct 30 22:44:47.991716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:47.991691    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6013c8-fwgfg"]
Oct 30 22:44:48.117026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:48.116903    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1397c723-9c26-42fa-b712-51f4aa20abc5" path="/var/lib/kubelet/pods/1397c723-9c26-42fa-b712-51f4aa20abc5/volumes"
Oct 30 22:44:50.223369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.223335    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"]
Oct 30 22:44:50.224762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.224740    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0899aba0-a556-4a34-a0db-d6205a977d62" containerName="payload-pre-processing"
Oct 30 22:44:50.224913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.224899    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0899aba0-a556-4a34-a0db-d6205a977d62" containerName="payload-pre-processing"
Oct 30 22:44:50.225014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.225005    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3f20acdb-7d98-459b-b668-91f1d4784470" containerName="payload-processing"
Oct 30 22:44:50.225089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.225080    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f20acdb-7d98-459b-b668-91f1d4784470" containerName="payload-processing"
Oct 30 22:44:50.225188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.225179    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1397c723-9c26-42fa-b712-51f4aa20abc5" containerName="revoke-keys"
Oct 30 22:44:50.225288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.225276    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1397c723-9c26-42fa-b712-51f4aa20abc5" containerName="revoke-keys"
Oct 30 22:44:50.225636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.225620    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1397c723-9c26-42fa-b712-51f4aa20abc5" containerName="revoke-keys"
Oct 30 22:44:50.225769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.225759    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0899aba0-a556-4a34-a0db-d6205a977d62" containerName="payload-pre-processing"
Oct 30 22:44:50.225849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.225841    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f20acdb-7d98-459b-b668-91f1d4784470" containerName="payload-processing"
Oct 30 22:44:50.247096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.247070    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.256336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.256294    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"]
Oct 30 22:44:50.304614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.304572    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bp5\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-kube-api-access-w6bp5\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.304811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.304699    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.304893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.304805    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-credential-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.304951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.304929    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-certs\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.305059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.305043    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-data\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.305125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.305087    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-podinfo\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.305263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.305240    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e171f96-3c69-4a25-ab53-f9112d808eb2-istiod-ca-cert\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.305347 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.305330    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-envoy\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.305403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.305367    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-token\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.352628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.352600    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-0ff5b0-openshift-default-dockercfg-2wvx5\""
Oct 30 22:44:50.407391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407360    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bp5\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-kube-api-access-w6bp5\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.407594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407413    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.407594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407452    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-credential-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.407594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407498    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-certs\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.407594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407582    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-data\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.407838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407631    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-podinfo\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.407838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407706    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e171f96-3c69-4a25-ab53-f9112d808eb2-istiod-ca-cert\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.407838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407782    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-envoy\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.407838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407815    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-token\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.408083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407886    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.408083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.407943    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-credential-socket\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.408346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.408323    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-data\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.408419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.408384    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-certs\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.408910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.408885    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e171f96-3c69-4a25-ab53-f9112d808eb2-istiod-ca-cert\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.410495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.410471    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-envoy\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.410608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.410520    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-podinfo\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.426159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.426129    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bp5\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-kube-api-access-w6bp5\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.426308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.426196    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-token\") pod \"e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") " pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.575913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.575835    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:50.918176 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.918143    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"]
Oct 30 22:44:50.919365 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:44:50.919339    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e171f96_3c69_4a25_ab53_f9112d808eb2.slice/crio-fe6d00a68b9c70343b89dffab9dbe73fc3571fe130358d14b1b4fcaea86d64b1 WatchSource:0}: Error finding container fe6d00a68b9c70343b89dffab9dbe73fc3571fe130358d14b1b4fcaea86d64b1: Status 404 returned error can't find the container with id fe6d00a68b9c70343b89dffab9dbe73fc3571fe130358d14b1b4fcaea86d64b1
Oct 30 22:44:50.922887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.922851    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:50.922985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.922921    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:50.922985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:50.922971    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:44:51.529397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:51.529352    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" event={"ID":"6e171f96-3c69-4a25-ab53-f9112d808eb2","Type":"ContainerStarted","Data":"49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7"}
Oct 30 22:44:51.529954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:51.529424    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" event={"ID":"6e171f96-3c69-4a25-ab53-f9112d808eb2","Type":"ContainerStarted","Data":"fe6d00a68b9c70343b89dffab9dbe73fc3571fe130358d14b1b4fcaea86d64b1"}
Oct 30 22:44:51.580276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:51.578326    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:51.593498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:51.593435    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" podStartSLOduration=1.5934174479999998 podStartE2EDuration="1.593417448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:51.59248766 +0000 UTC m=+2275.631426545" watchObservedRunningTime="2026-10-30 22:44:51.593417448 +0000 UTC m=+2275.632356338"
Oct 30 22:44:52.580399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:52.580351    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:44:52.580943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:52.580420    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 22:44:53.581255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:53.578517    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:44:53.581255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:53.578576    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 22:44:54.576826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:54.576779    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:44:54.577040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:54.576852    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 22:44:55.104765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:55.104726    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"]
Oct 30 22:44:55.105598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:55.105563    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy" containerID="cri-o://aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209" gracePeriod=30
Oct 30 22:44:55.576817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:55.576723    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:44:55.577096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:55.577068    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 22:44:56.578244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:56.577608    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:44:56.578244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:56.577686    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 22:44:57.577698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:57.577662    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:44:57.577886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:57.577740    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 22:44:58.581835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:58.581800    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:58.582391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:58.582252    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:44:58.583025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:44:58.583007    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:45:00.166690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.166652    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"]
Oct 30 22:45:00.174627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.174590    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"]
Oct 30 22:45:00.174807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.174732    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:00.180563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.180532    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"]
Oct 30 22:45:00.180770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.180653    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:00.186449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.186419    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:00.186618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.186419    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"]
Oct 30 22:45:00.188352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.188325    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:00.192149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.192122    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:00.192324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.192123    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:00.192455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.192437    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:45:00.192883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.192860    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"]
Oct 30 22:45:00.198773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.198746    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:00.201502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.201479    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"]
Oct 30 22:45:00.208669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.208645    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:00.209175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.209158    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"]
Oct 30 22:45:00.219924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.219896    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"]
Oct 30 22:45:00.220319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.220299    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:00.259639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.226338    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"]
Oct 30 22:45:00.259639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.230008    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:00.275790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.275640    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:00.282885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.282853    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc\" (UID: \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:00.283027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.282888    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abeab3fd-da86-4674-943d-0560abd2e752-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz\" (UID: \"abeab3fd-da86-4674-943d-0560abd2e752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:00.283098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.283046    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-kube-api-access-m5xtp\") pod \"maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc\" (UID: \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:00.283098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.283073    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93290c9-59d9-48e6-990a-b18d680913ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq\" (UID: \"e93290c9-59d9-48e6-990a-b18d680913ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:00.283605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.283544    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj75m\" (UniqueName: \"kubernetes.io/projected/e93290c9-59d9-48e6-990a-b18d680913ed-kube-api-access-kj75m\") pod \"maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq\" (UID: \"e93290c9-59d9-48e6-990a-b18d680913ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:00.283763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.283739    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fls6n\" (UniqueName: \"kubernetes.io/projected/abeab3fd-da86-4674-943d-0560abd2e752-kube-api-access-fls6n\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz\" (UID: \"abeab3fd-da86-4674-943d-0560abd2e752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:00.359690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.338303    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"]
Oct 30 22:45:00.359690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.344797    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"]
Oct 30 22:45:00.359690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.352657    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"]
Oct 30 22:45:00.361505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.361480    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"]
Oct 30 22:45:00.368492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.368466    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"]
Oct 30 22:45:00.382424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.382394    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"]
Oct 30 22:45:00.386071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386043    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0420e-107e-40c7-8d92-85c50abb696d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw\" (UID: \"23d0420e-107e-40c7-8d92-85c50abb696d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:00.386208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386087    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvmc\" (UniqueName: \"kubernetes.io/projected/fd40171c-a1f5-484c-a078-ce72ef26cb90-kube-api-access-7kvmc\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d\" (UID: \"fd40171c-a1f5-484c-a078-ce72ef26cb90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:00.386208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386146    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dnq\" (UniqueName: \"kubernetes.io/projected/decc6e06-9261-4213-a87b-26fb98bcbe50-kube-api-access-d2dnq\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv\" (UID: \"decc6e06-9261-4213-a87b-26fb98bcbe50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:00.386208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386187    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f7488c-2538-4dc6-818b-306e295933cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw\" (UID: \"a5f7488c-2538-4dc6-818b-306e295933cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:00.386424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386244    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fls6n\" (UniqueName: \"kubernetes.io/projected/abeab3fd-da86-4674-943d-0560abd2e752-kube-api-access-fls6n\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz\" (UID: \"abeab3fd-da86-4674-943d-0560abd2e752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:00.386424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386306    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc\" (UID: \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:00.386424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386334    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abeab3fd-da86-4674-943d-0560abd2e752-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz\" (UID: \"abeab3fd-da86-4674-943d-0560abd2e752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:00.386424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386384    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c00be-1d02-4314-83bd-d27450715d23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm\" (UID: \"169c00be-1d02-4314-83bd-d27450715d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:00.386636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386486    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decc6e06-9261-4213-a87b-26fb98bcbe50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv\" (UID: \"decc6e06-9261-4213-a87b-26fb98bcbe50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:00.386636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386542    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-kube-api-access-m5xtp\") pod \"maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc\" (UID: \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:00.386636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386573    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93290c9-59d9-48e6-990a-b18d680913ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq\" (UID: \"e93290c9-59d9-48e6-990a-b18d680913ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:00.386636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386612    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svzx4\" (UniqueName: \"kubernetes.io/projected/169c00be-1d02-4314-83bd-d27450715d23-kube-api-access-svzx4\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm\" (UID: \"169c00be-1d02-4314-83bd-d27450715d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:00.386851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386767    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgf6g\" (UniqueName: \"kubernetes.io/projected/fb334b3a-3f29-4505-ae06-d4ce0917bebc-kube-api-access-wgf6g\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd\" (UID: \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:00.386851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386798    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb334b3a-3f29-4505-ae06-d4ce0917bebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd\" (UID: \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:00.386963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386850    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7276\" (UniqueName: \"kubernetes.io/projected/a5f7488c-2538-4dc6-818b-306e295933cc-kube-api-access-j7276\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw\" (UID: \"a5f7488c-2538-4dc6-818b-306e295933cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:00.386963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386914    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkvcz\" (UniqueName: \"kubernetes.io/projected/23d0420e-107e-40c7-8d92-85c50abb696d-kube-api-access-vkvcz\") pod \"maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw\" (UID: \"23d0420e-107e-40c7-8d92-85c50abb696d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:00.386963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.386957    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd40171c-a1f5-484c-a078-ce72ef26cb90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d\" (UID: \"fd40171c-a1f5-484c-a078-ce72ef26cb90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:00.387121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.387037    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj75m\" (UniqueName: \"kubernetes.io/projected/e93290c9-59d9-48e6-990a-b18d680913ed-kube-api-access-kj75m\") pod \"maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq\" (UID: \"e93290c9-59d9-48e6-990a-b18d680913ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:00.388055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.388028    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc\" (UID: \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:00.388635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.388581    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abeab3fd-da86-4674-943d-0560abd2e752-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz\" (UID: \"abeab3fd-da86-4674-943d-0560abd2e752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:00.389129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.389106    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93290c9-59d9-48e6-990a-b18d680913ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq\" (UID: \"e93290c9-59d9-48e6-990a-b18d680913ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:00.393194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.391840    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"]
Oct 30 22:45:00.394304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.394284    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:00.399796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.399776    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"]
Oct 30 22:45:00.399956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.399862    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:00.403386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.403365    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"]
Oct 30 22:45:00.414192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.411626    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"]
Oct 30 22:45:00.421575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.414833    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fls6n\" (UniqueName: \"kubernetes.io/projected/abeab3fd-da86-4674-943d-0560abd2e752-kube-api-access-fls6n\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz\" (UID: \"abeab3fd-da86-4674-943d-0560abd2e752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:00.421575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.418564    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj75m\" (UniqueName: \"kubernetes.io/projected/e93290c9-59d9-48e6-990a-b18d680913ed-kube-api-access-kj75m\") pod \"maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq\" (UID: \"e93290c9-59d9-48e6-990a-b18d680913ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:00.421575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.420964    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-kube-api-access-m5xtp\") pod \"maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc\" (UID: \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:00.439887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.433472    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"]
Oct 30 22:45:00.439887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.433589    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:00.447981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.447941    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:00.458199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.458167    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"]
Oct 30 22:45:00.469042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.469004    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"]
Oct 30 22:45:00.489659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.489624    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0420e-107e-40c7-8d92-85c50abb696d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw\" (UID: \"23d0420e-107e-40c7-8d92-85c50abb696d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:00.489917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.489896    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvmc\" (UniqueName: \"kubernetes.io/projected/fd40171c-a1f5-484c-a078-ce72ef26cb90-kube-api-access-7kvmc\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d\" (UID: \"fd40171c-a1f5-484c-a078-ce72ef26cb90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:00.490084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.490065    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dnq\" (UniqueName: \"kubernetes.io/projected/decc6e06-9261-4213-a87b-26fb98bcbe50-kube-api-access-d2dnq\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv\" (UID: \"decc6e06-9261-4213-a87b-26fb98bcbe50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:00.490252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.490236    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f7488c-2538-4dc6-818b-306e295933cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw\" (UID: \"a5f7488c-2538-4dc6-818b-306e295933cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:00.490411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.490392    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqbsx\" (UniqueName: \"kubernetes.io/projected/071439a8-bb97-4ab1-b4af-e0c88e7a585e-kube-api-access-cqbsx\") pod \"maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc\" (UID: \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:00.490603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.490584    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c00be-1d02-4314-83bd-d27450715d23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm\" (UID: \"169c00be-1d02-4314-83bd-d27450715d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:00.490823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.490808    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decc6e06-9261-4213-a87b-26fb98bcbe50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv\" (UID: \"decc6e06-9261-4213-a87b-26fb98bcbe50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:00.490966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.490951    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/071439a8-bb97-4ab1-b4af-e0c88e7a585e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc\" (UID: \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:00.491730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.491700    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svzx4\" (UniqueName: \"kubernetes.io/projected/169c00be-1d02-4314-83bd-d27450715d23-kube-api-access-svzx4\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm\" (UID: \"169c00be-1d02-4314-83bd-d27450715d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:00.491882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.491866    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgf6g\" (UniqueName: \"kubernetes.io/projected/fb334b3a-3f29-4505-ae06-d4ce0917bebc-kube-api-access-wgf6g\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd\" (UID: \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:00.491957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.491908    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb334b3a-3f29-4505-ae06-d4ce0917bebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd\" (UID: \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:00.491957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.491946    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f7488c-2538-4dc6-818b-306e295933cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw\" (UID: \"a5f7488c-2538-4dc6-818b-306e295933cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:00.492084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.491958    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7276\" (UniqueName: \"kubernetes.io/projected/a5f7488c-2538-4dc6-818b-306e295933cc-kube-api-access-j7276\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw\" (UID: \"a5f7488c-2538-4dc6-818b-306e295933cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:00.492084 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.492067    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decc6e06-9261-4213-a87b-26fb98bcbe50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv\" (UID: \"decc6e06-9261-4213-a87b-26fb98bcbe50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:00.493500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.492631    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0420e-107e-40c7-8d92-85c50abb696d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw\" (UID: \"23d0420e-107e-40c7-8d92-85c50abb696d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:00.493500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.492948    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb334b3a-3f29-4505-ae06-d4ce0917bebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd\" (UID: \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:00.493500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.493001    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkvcz\" (UniqueName: \"kubernetes.io/projected/23d0420e-107e-40c7-8d92-85c50abb696d-kube-api-access-vkvcz\") pod \"maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw\" (UID: \"23d0420e-107e-40c7-8d92-85c50abb696d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:00.493500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.493054    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd40171c-a1f5-484c-a078-ce72ef26cb90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d\" (UID: \"fd40171c-a1f5-484c-a078-ce72ef26cb90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:00.493856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.493827    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd40171c-a1f5-484c-a078-ce72ef26cb90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d\" (UID: \"fd40171c-a1f5-484c-a078-ce72ef26cb90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:00.494334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.494312    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c00be-1d02-4314-83bd-d27450715d23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm\" (UID: \"169c00be-1d02-4314-83bd-d27450715d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:00.512060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.512028    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"]
Oct 30 22:45:00.514562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.512479    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:00.514819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.514799    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"]
Oct 30 22:45:00.516874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.516841    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7276\" (UniqueName: \"kubernetes.io/projected/a5f7488c-2538-4dc6-818b-306e295933cc-kube-api-access-j7276\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw\" (UID: \"a5f7488c-2538-4dc6-818b-306e295933cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:00.518925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.517716    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkvcz\" (UniqueName: \"kubernetes.io/projected/23d0420e-107e-40c7-8d92-85c50abb696d-kube-api-access-vkvcz\") pod \"maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw\" (UID: \"23d0420e-107e-40c7-8d92-85c50abb696d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:00.520966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.519541    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dnq\" (UniqueName: \"kubernetes.io/projected/decc6e06-9261-4213-a87b-26fb98bcbe50-kube-api-access-d2dnq\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv\" (UID: \"decc6e06-9261-4213-a87b-26fb98bcbe50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:00.520966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.519591    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svzx4\" (UniqueName: \"kubernetes.io/projected/169c00be-1d02-4314-83bd-d27450715d23-kube-api-access-svzx4\") pod \"maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm\" (UID: \"169c00be-1d02-4314-83bd-d27450715d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:00.520966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.520201    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvmc\" (UniqueName: \"kubernetes.io/projected/fd40171c-a1f5-484c-a078-ce72ef26cb90-kube-api-access-7kvmc\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d\" (UID: \"fd40171c-a1f5-484c-a078-ce72ef26cb90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:00.522469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.522163    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgf6g\" (UniqueName: \"kubernetes.io/projected/fb334b3a-3f29-4505-ae06-d4ce0917bebc-kube-api-access-wgf6g\") pod \"maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd\" (UID: \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:00.526942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.526922    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"]
Oct 30 22:45:00.534280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.534256    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:00.534481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.534467    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"]
Oct 30 22:45:00.535612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.535587    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:00.540518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.539898    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:00.547412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.547390    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"]
Oct 30 22:45:00.553791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.550545    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:00.557295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.557274    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"]
Oct 30 22:45:00.564036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.564015    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"]
Oct 30 22:45:00.572371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.572345    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"]
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.589358    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"]
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.596554    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787sf\" (UniqueName: \"kubernetes.io/projected/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-kube-api-access-787sf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85\" (UID: \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.596810    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbsx\" (UniqueName: \"kubernetes.io/projected/071439a8-bb97-4ab1-b4af-e0c88e7a585e-kube-api-access-cqbsx\") pod \"maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc\" (UID: \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.596993    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/071439a8-bb97-4ab1-b4af-e0c88e7a585e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc\" (UID: \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.597032    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f18846-74ba-48c9-a797-0f33d1820e16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8\" (UID: \"31f18846-74ba-48c9-a797-0f33d1820e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.597094    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85\" (UID: \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.597121    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhfng\" (UniqueName: \"kubernetes.io/projected/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-kube-api-access-qhfng\") pod \"maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v\" (UID: \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.597192    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkxr\" (UniqueName: \"kubernetes.io/projected/31f18846-74ba-48c9-a797-0f33d1820e16-kube-api-access-7fkxr\") pod \"maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8\" (UID: \"31f18846-74ba-48c9-a797-0f33d1820e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.597994    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/071439a8-bb97-4ab1-b4af-e0c88e7a585e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc\" (UID: \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:00.599245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.598023    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v\" (UID: \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:00.600406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.600381    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"]
Oct 30 22:45:00.600536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.600474    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:00.618662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.607820    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"]
Oct 30 22:45:00.618662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.614958    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbsx\" (UniqueName: \"kubernetes.io/projected/071439a8-bb97-4ab1-b4af-e0c88e7a585e-kube-api-access-cqbsx\") pod \"maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc\" (UID: \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:00.618662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.615617    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:45:00.622133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.622093    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:00.631201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.631170    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"]
Oct 30 22:45:00.634794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.634768    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:00.645265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.644864    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:00.664043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.664006    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:00.682876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.682187    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:00.700085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.699254    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:00.700085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.700005    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kldr7\" (UniqueName: \"kubernetes.io/projected/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-kube-api-access-kldr7\") pod \"maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5\" (UID: \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:00.700329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.700160    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8dz2\" (UniqueName: \"kubernetes.io/projected/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-kube-api-access-t8dz2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw\" (UID: \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:00.700329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.700261    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5\" (UID: \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.700339    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f18846-74ba-48c9-a797-0f33d1820e16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8\" (UID: \"31f18846-74ba-48c9-a797-0f33d1820e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.700888    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7g7b\" (UniqueName: \"kubernetes.io/projected/abfeb292-6e71-4543-9215-a445f8226c13-kube-api-access-x7g7b\") pod \"maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b\" (UID: \"abfeb292-6e71-4543-9215-a445f8226c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.700922    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr\" (UID: \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.700972    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85\" (UID: \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.701002    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhfng\" (UniqueName: \"kubernetes.io/projected/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-kube-api-access-qhfng\") pod \"maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v\" (UID: \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.701058    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbls7\" (UniqueName: \"kubernetes.io/projected/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-kube-api-access-dbls7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr\" (UID: \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.701100    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkxr\" (UniqueName: \"kubernetes.io/projected/31f18846-74ba-48c9-a797-0f33d1820e16-kube-api-access-7fkxr\") pod \"maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8\" (UID: \"31f18846-74ba-48c9-a797-0f33d1820e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.701160    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v\" (UID: \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.701248    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfeb292-6e71-4543-9215-a445f8226c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b\" (UID: \"abfeb292-6e71-4543-9215-a445f8226c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.701293    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw\" (UID: \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.701365    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787sf\" (UniqueName: \"kubernetes.io/projected/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-kube-api-access-787sf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85\" (UID: \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.702506    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f18846-74ba-48c9-a797-0f33d1820e16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8\" (UID: \"31f18846-74ba-48c9-a797-0f33d1820e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.703412    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85\" (UID: \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:00.720048 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.704366    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v\" (UID: \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:00.728563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.727730    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-787sf\" (UniqueName: \"kubernetes.io/projected/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-kube-api-access-787sf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85\" (UID: \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:00.730544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.729063    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkxr\" (UniqueName: \"kubernetes.io/projected/31f18846-74ba-48c9-a797-0f33d1820e16-kube-api-access-7fkxr\") pod \"maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8\" (UID: \"31f18846-74ba-48c9-a797-0f33d1820e16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:00.740592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.739897    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhfng\" (UniqueName: \"kubernetes.io/projected/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-kube-api-access-qhfng\") pod \"maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v\" (UID: \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:00.765458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.765263    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:00.798627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.785410    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.802351    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-credential-socket\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.802426    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-podinfo\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.802501    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-socket\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.802630    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-token\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-token\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.802700    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istiod-ca-cert\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.802809    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-data\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.802840    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-envoy\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.802907    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-certs\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.803006    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-kube-api-access-sxlvz\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-kube-api-access-sxlvz\") pod \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\" (UID: \"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9\") "
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.803415    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7g7b\" (UniqueName: \"kubernetes.io/projected/abfeb292-6e71-4543-9215-a445f8226c13-kube-api-access-x7g7b\") pod \"maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b\" (UID: \"abfeb292-6e71-4543-9215-a445f8226c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.803451    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr\" (UID: \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.803538    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbls7\" (UniqueName: \"kubernetes.io/projected/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-kube-api-access-dbls7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr\" (UID: \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.803666    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfeb292-6e71-4543-9215-a445f8226c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b\" (UID: \"abfeb292-6e71-4543-9215-a445f8226c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.803715    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw\" (UID: \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.803866    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kldr7\" (UniqueName: \"kubernetes.io/projected/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-kube-api-access-kldr7\") pod \"maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5\" (UID: \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.804030    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8dz2\" (UniqueName: \"kubernetes.io/projected/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-kube-api-access-t8dz2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw\" (UID: \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.804124    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5\" (UID: \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.805023    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5\" (UID: \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.805604    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-credential-socket" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.806592    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr\" (UID: \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.807844    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istiod-ca-cert" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.809101    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-socket" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.809369    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-data" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.809591    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-certs" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.810997    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.811025    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw\" (UID: \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.811469    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfeb292-6e71-4543-9215-a445f8226c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b\" (UID: \"abfeb292-6e71-4543-9215-a445f8226c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:00.811651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.811617    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-kube-api-access-sxlvz" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "kube-api-access-sxlvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:00.817196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.817166    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-envoy" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:00.818754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.818718    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-podinfo" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:45:00.818980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.818960    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-token" pod "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" (UID: "636c8afd-b441-43f0-b5fb-ddc2a33dd2c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:00.829419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.824977    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7g7b\" (UniqueName: \"kubernetes.io/projected/abfeb292-6e71-4543-9215-a445f8226c13-kube-api-access-x7g7b\") pod \"maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b\" (UID: \"abfeb292-6e71-4543-9215-a445f8226c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:00.829419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.825084    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbls7\" (UniqueName: \"kubernetes.io/projected/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-kube-api-access-dbls7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr\" (UID: \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:00.829419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.828149    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8dz2\" (UniqueName: \"kubernetes.io/projected/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-kube-api-access-t8dz2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw\" (UID: \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:00.837791 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.837728    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kldr7\" (UniqueName: \"kubernetes.io/projected/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-kube-api-access-kldr7\") pod \"maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5\" (UID: \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:00.878682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.877899    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:00.899158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.899120    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"]
Oct 30 22:45:00.915279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.901494    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917473    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917508    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917521    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917535    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917546    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917560    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxlvz\" (UniqueName: \"kubernetes.io/projected/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-kube-api-access-sxlvz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917577    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917592    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.918077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.917606    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:00.971365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.971332    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:00.982399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:00.982349    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"]
Oct 30 22:45:01.007992 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.007952    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabeab3fd_da86_4674_943d_0560abd2e752.slice/crio-c4293e3fa3220d8947f8b4df62e76cbd812ccbf71ca8f924bfe1fd26d985606b WatchSource:0}: Error finding container c4293e3fa3220d8947f8b4df62e76cbd812ccbf71ca8f924bfe1fd26d985606b: Status 404 returned error can't find the container with id c4293e3fa3220d8947f8b4df62e76cbd812ccbf71ca8f924bfe1fd26d985606b
Oct 30 22:45:01.027658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.027616    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:01.066844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.066358    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:01.084635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.083436    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:01.316447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.315441    4581 generic.go:356] "Generic (PLEG): container finished" podID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerID="aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209" exitCode=0
Oct 30 22:45:01.316447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.315753    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" event={"ID":"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9","Type":"ContainerDied","Data":"aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209"}
Oct 30 22:45:01.316447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.315985    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" event={"ID":"636c8afd-b441-43f0-b5fb-ddc2a33dd2c9","Type":"ContainerDied","Data":"ca123f88759d85f55fb31283aa10ebb4edeecd979a9dde175a5491c1f5bc81f5"}
Oct 30 22:45:01.316447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.316045    4581 scope.go:122] "RemoveContainer" containerID="aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209"
Oct 30 22:45:01.318964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.317392    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"
Oct 30 22:45:01.339135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.339106    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"]
Oct 30 22:45:01.461530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.461379    4581 scope.go:122] "RemoveContainer" containerID="305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469"
Oct 30 22:45:01.463547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.463368    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"]
Oct 30 22:45:01.472600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.472551    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc"]
Oct 30 22:45:01.569020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.566649    4581 scope.go:122] "RemoveContainer" containerID="aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209"
Oct 30 22:45:01.569020 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:01.568115    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209\": container with ID starting with aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209 not found: ID does not exist" containerID="aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209"
Oct 30 22:45:01.569020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.568156    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209"} err="failed to get container status \"aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209\": rpc error: code = NotFound desc = could not find container \"aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209\": container with ID starting with aeb1a62b40528546b2d9f520eb8ddf0d8798dc7b9f74168ddb549b68f1484209 not found: ID does not exist"
Oct 30 22:45:01.569020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.568181    4581 scope.go:122] "RemoveContainer" containerID="305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469"
Oct 30 22:45:01.569020 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:01.568585    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469\": container with ID starting with 305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469 not found: ID does not exist" containerID="305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469"
Oct 30 22:45:01.569020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.568614    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469"} err="failed to get container status \"305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469\": rpc error: code = NotFound desc = could not find container \"305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469\": container with ID starting with 305c8a8e803ca8ec0a950bf391c8eb03babb161b970569ad4360d00c4df7c469 not found: ID does not exist"
Oct 30 22:45:01.587008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.583321    4581 patch_prober.go:29] interesting pod/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:45:01.587008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.583386    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6013c8-openshift-default-96f465855-llgxc" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:45:01.633470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.633440    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"]
Oct 30 22:45:01.634550 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.634518    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd40171c_a1f5_484c_a078_ce72ef26cb90.slice/crio-ab2bd63c8740e35150b93e570a1ea025eedced838d1a6f35852cf2dcd283eedc WatchSource:0}: Error finding container ab2bd63c8740e35150b93e570a1ea025eedced838d1a6f35852cf2dcd283eedc: Status 404 returned error can't find the container with id ab2bd63c8740e35150b93e570a1ea025eedced838d1a6f35852cf2dcd283eedc
Oct 30 22:45:01.661154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.661121    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"]
Oct 30 22:45:01.676893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.676859    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"]
Oct 30 22:45:01.771400 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.771371    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecc6e06_9261_4213_a87b_26fb98bcbe50.slice/crio-421e5b7b18bda526bdd45bf3e5700fe31793fb1c02e6b19a3a9bb2bed28843af WatchSource:0}: Error finding container 421e5b7b18bda526bdd45bf3e5700fe31793fb1c02e6b19a3a9bb2bed28843af: Status 404 returned error can't find the container with id 421e5b7b18bda526bdd45bf3e5700fe31793fb1c02e6b19a3a9bb2bed28843af
Oct 30 22:45:01.867780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.867631    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"]
Oct 30 22:45:01.873832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.873803    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"]
Oct 30 22:45:01.888957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.888923    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"]
Oct 30 22:45:01.908871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.908838    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"]
Oct 30 22:45:01.917569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.917541    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"]
Oct 30 22:45:01.928317 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.928292    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"]
Oct 30 22:45:01.940008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.939952    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"]
Oct 30 22:45:01.945458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:01.945438    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"]
Oct 30 22:45:01.958921 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.958893    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071439a8_bb97_4ab1_b4af_e0c88e7a585e.slice/crio-ce5bb3635ae9b6ec789f6299ceb7d2e5b21d0a637201f4f6a5ff53459bdbe239 WatchSource:0}: Error finding container ce5bb3635ae9b6ec789f6299ceb7d2e5b21d0a637201f4f6a5ff53459bdbe239: Status 404 returned error can't find the container with id ce5bb3635ae9b6ec789f6299ceb7d2e5b21d0a637201f4f6a5ff53459bdbe239
Oct 30 22:45:01.963009 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.962970    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169c00be_1d02_4314_83bd_d27450715d23.slice/crio-9345844b574c735f40694938bd6fcccf62e6b0b5cd03e24b1a92bd94dd16b394 WatchSource:0}: Error finding container 9345844b574c735f40694938bd6fcccf62e6b0b5cd03e24b1a92bd94dd16b394: Status 404 returned error can't find the container with id 9345844b574c735f40694938bd6fcccf62e6b0b5cd03e24b1a92bd94dd16b394
Oct 30 22:45:01.964757 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.964735    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2296d534_4a85_4b40_a54e_c8b2a67e2aa2.slice/crio-29749b6c02842061d204f8669e398875b6cf511e804c7d27882a84da63de64ad WatchSource:0}: Error finding container 29749b6c02842061d204f8669e398875b6cf511e804c7d27882a84da63de64ad: Status 404 returned error can't find the container with id 29749b6c02842061d204f8669e398875b6cf511e804c7d27882a84da63de64ad
Oct 30 22:45:01.965331 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.965304    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfeb292_6e71_4543_9215_a445f8226c13.slice/crio-e552ae831591ad2a8986bddc593ed0e8a0fc0a9aae9ca07f2e3c3b372c9f1586 WatchSource:0}: Error finding container e552ae831591ad2a8986bddc593ed0e8a0fc0a9aae9ca07f2e3c3b372c9f1586: Status 404 returned error can't find the container with id e552ae831591ad2a8986bddc593ed0e8a0fc0a9aae9ca07f2e3c3b372c9f1586
Oct 30 22:45:01.967439 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.967416    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d0420e_107e_40c7_8d92_85c50abb696d.slice/crio-e60d4a05a243a1a4d88bc2161464e159d4c013dd2723f9976b16c4e9954e8dda WatchSource:0}: Error finding container e60d4a05a243a1a4d88bc2161464e159d4c013dd2723f9976b16c4e9954e8dda: Status 404 returned error can't find the container with id e60d4a05a243a1a4d88bc2161464e159d4c013dd2723f9976b16c4e9954e8dda
Oct 30 22:45:01.969156 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.969134    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3f411a_89df_4177_bb36_b9bdcca5e8c2.slice/crio-bccdf55a2ab4b23d940da03b9c7ed81ace0822f51f63db40d4e99cf52e64abd8 WatchSource:0}: Error finding container bccdf55a2ab4b23d940da03b9c7ed81ace0822f51f63db40d4e99cf52e64abd8: Status 404 returned error can't find the container with id bccdf55a2ab4b23d940da03b9c7ed81ace0822f51f63db40d4e99cf52e64abd8
Oct 30 22:45:01.969585 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:01.969563    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb334b3a_3f29_4505_ae06_d4ce0917bebc.slice/crio-fd81750b24b1e087b8adbe21d001071efdfe980dab02ee789b72d28ec4e28a4b WatchSource:0}: Error finding container fd81750b24b1e087b8adbe21d001071efdfe980dab02ee789b72d28ec4e28a4b: Status 404 returned error can't find the container with id fd81750b24b1e087b8adbe21d001071efdfe980dab02ee789b72d28ec4e28a4b
Oct 30 22:45:02.005015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.004985    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"]
Oct 30 22:45:02.007392 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:02.007360    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f18846_74ba_48c9_a797_0f33d1820e16.slice/crio-d6be1c9d47fcf84771833730596193283c5c542f70f27697105afc2f085ac23c WatchSource:0}: Error finding container d6be1c9d47fcf84771833730596193283c5c542f70f27697105afc2f085ac23c: Status 404 returned error can't find the container with id d6be1c9d47fcf84771833730596193283c5c542f70f27697105afc2f085ac23c
Oct 30 22:45:02.049471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.049445    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"]
Oct 30 22:45:02.066960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.064551    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"]
Oct 30 22:45:02.102458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.102421    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" path="/var/lib/kubelet/pods/636c8afd-b441-43f0-b5fb-ddc2a33dd2c9/volumes"
Oct 30 22:45:02.256684 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:02.256646    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9af81a_12ae_4b6d_8d0a_8a90c3b11ba3.slice/crio-22b297a97760762c6b7397b94d0331ef138b5228687f7607f22e23028edee424 WatchSource:0}: Error finding container 22b297a97760762c6b7397b94d0331ef138b5228687f7607f22e23028edee424: Status 404 returned error can't find the container with id 22b297a97760762c6b7397b94d0331ef138b5228687f7607f22e23028edee424
Oct 30 22:45:02.328041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.328001    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" event={"ID":"e93290c9-59d9-48e6-990a-b18d680913ed","Type":"ContainerStarted","Data":"a1160ddde82238e1631478f39b64f0116a7f5c779db3a7d6bc28f034eaddadf3"}
Oct 30 22:45:02.329446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.329419    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" event={"ID":"169c00be-1d02-4314-83bd-d27450715d23","Type":"ContainerStarted","Data":"9345844b574c735f40694938bd6fcccf62e6b0b5cd03e24b1a92bd94dd16b394"}
Oct 30 22:45:02.331208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.331173    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" event={"ID":"abfeb292-6e71-4543-9215-a445f8226c13","Type":"ContainerStarted","Data":"e552ae831591ad2a8986bddc593ed0e8a0fc0a9aae9ca07f2e3c3b372c9f1586"}
Oct 30 22:45:02.333728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.333697    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" event={"ID":"071439a8-bb97-4ab1-b4af-e0c88e7a585e","Type":"ContainerStarted","Data":"ce5bb3635ae9b6ec789f6299ceb7d2e5b21d0a637201f4f6a5ff53459bdbe239"}
Oct 30 22:45:02.335272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.335208    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" event={"ID":"31f18846-74ba-48c9-a797-0f33d1820e16","Type":"ContainerStarted","Data":"d6be1c9d47fcf84771833730596193283c5c542f70f27697105afc2f085ac23c"}
Oct 30 22:45:02.336703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.336684    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" event={"ID":"dd3f411a-89df-4177-bb36-b9bdcca5e8c2","Type":"ContainerStarted","Data":"bccdf55a2ab4b23d940da03b9c7ed81ace0822f51f63db40d4e99cf52e64abd8"}
Oct 30 22:45:02.338075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.338047    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" event={"ID":"a5f7488c-2538-4dc6-818b-306e295933cc","Type":"ContainerStarted","Data":"188a4b75c050d0d957bd12a03a7e88430a56347af1e4c39c0b47700663d493b7"}
Oct 30 22:45:02.339258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.339236    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" event={"ID":"fb334b3a-3f29-4505-ae06-d4ce0917bebc","Type":"ContainerStarted","Data":"fd81750b24b1e087b8adbe21d001071efdfe980dab02ee789b72d28ec4e28a4b"}
Oct 30 22:45:02.340446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.340427    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" event={"ID":"abeab3fd-da86-4674-943d-0560abd2e752","Type":"ContainerStarted","Data":"c4293e3fa3220d8947f8b4df62e76cbd812ccbf71ca8f924bfe1fd26d985606b"}
Oct 30 22:45:02.342125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.342067    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" event={"ID":"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb","Type":"ContainerStarted","Data":"bf75fe4bc38224d477f5a48c645fc7586f571273e216768bd9025a0da4b7cf29"}
Oct 30 22:45:02.342125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.342111    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" event={"ID":"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb","Type":"ContainerStarted","Data":"a1f8f72e411a4a891c035ee7e936012e7f4705748c9ef9cf6deb7a3172e5651f"}
Oct 30 22:45:02.343979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.343955    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" event={"ID":"23d0420e-107e-40c7-8d92-85c50abb696d","Type":"ContainerStarted","Data":"e60d4a05a243a1a4d88bc2161464e159d4c013dd2723f9976b16c4e9954e8dda"}
Oct 30 22:45:02.345453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.345429    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" event={"ID":"decc6e06-9261-4213-a87b-26fb98bcbe50","Type":"ContainerStarted","Data":"421e5b7b18bda526bdd45bf3e5700fe31793fb1c02e6b19a3a9bb2bed28843af"}
Oct 30 22:45:02.346793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.346771    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" event={"ID":"2296d534-4a85-4b40-a54e-c8b2a67e2aa2","Type":"ContainerStarted","Data":"29749b6c02842061d204f8669e398875b6cf511e804c7d27882a84da63de64ad"}
Oct 30 22:45:02.348115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.348096    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" event={"ID":"fd40171c-a1f5-484c-a078-ce72ef26cb90","Type":"ContainerStarted","Data":"ab2bd63c8740e35150b93e570a1ea025eedced838d1a6f35852cf2dcd283eedc"}
Oct 30 22:45:02.349367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.349344    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" event={"ID":"34eb7cbf-d1bd-4141-b0a1-6d7833377c53","Type":"ContainerStarted","Data":"c68772e8d6b2642c62ffff76abc68466c1bbe4f4301d9db5ca6979b1c81ad14a"}
Oct 30 22:45:02.350932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:02.350907    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" event={"ID":"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3","Type":"ContainerStarted","Data":"22b297a97760762c6b7397b94d0331ef138b5228687f7607f22e23028edee424"}
Oct 30 22:45:03.379483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.379446    4581 generic.go:356] "Generic (PLEG): container finished" podID="a5f7488c-2538-4dc6-818b-306e295933cc" containerID="952d949f5475809f88ac09eb9b08455e6c856add8e2c987db3804c0e617864e6" exitCode=60
Oct 30 22:45:03.379895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.379574    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" event={"ID":"a5f7488c-2538-4dc6-818b-306e295933cc","Type":"ContainerDied","Data":"952d949f5475809f88ac09eb9b08455e6c856add8e2c987db3804c0e617864e6"}
Oct 30 22:45:03.379895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.379882    4581 scope.go:122] "RemoveContainer" containerID="952d949f5475809f88ac09eb9b08455e6c856add8e2c987db3804c0e617864e6"
Oct 30 22:45:03.383383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.383166    4581 scope.go:122] "RemoveContainer" containerID="1c6ee7b7d3e182d67935d8c6d97aec2cbf3da0c50c1e5a8b0deacde3d333c3d0"
Oct 30 22:45:03.384779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.384289    4581 generic.go:356] "Generic (PLEG): container finished" podID="abeab3fd-da86-4674-943d-0560abd2e752" containerID="1c6ee7b7d3e182d67935d8c6d97aec2cbf3da0c50c1e5a8b0deacde3d333c3d0" exitCode=60
Oct 30 22:45:03.384779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.384388    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" event={"ID":"abeab3fd-da86-4674-943d-0560abd2e752","Type":"ContainerDied","Data":"1c6ee7b7d3e182d67935d8c6d97aec2cbf3da0c50c1e5a8b0deacde3d333c3d0"}
Oct 30 22:45:03.388622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.387910    4581 generic.go:356] "Generic (PLEG): container finished" podID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerID="bf75fe4bc38224d477f5a48c645fc7586f571273e216768bd9025a0da4b7cf29" exitCode=60
Oct 30 22:45:03.388622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.388006    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" event={"ID":"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb","Type":"ContainerDied","Data":"bf75fe4bc38224d477f5a48c645fc7586f571273e216768bd9025a0da4b7cf29"}
Oct 30 22:45:03.388622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.388320    4581 scope.go:122] "RemoveContainer" containerID="bf75fe4bc38224d477f5a48c645fc7586f571273e216768bd9025a0da4b7cf29"
Oct 30 22:45:03.391174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.390718    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" event={"ID":"23d0420e-107e-40c7-8d92-85c50abb696d","Type":"ContainerStarted","Data":"148e0a493a0da0fc06704b7324aee18dc63ca0753ab6dbbc9b9f216bc9211fc8"}
Oct 30 22:45:03.393476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.393455    4581 generic.go:356] "Generic (PLEG): container finished" podID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerID="8ff52e25dbf3cf134e311181026f9fd71a3ec77259c2b43e5b2fe2ad94ca28b3" exitCode=60
Oct 30 22:45:03.393578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.393557    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" event={"ID":"decc6e06-9261-4213-a87b-26fb98bcbe50","Type":"ContainerDied","Data":"8ff52e25dbf3cf134e311181026f9fd71a3ec77259c2b43e5b2fe2ad94ca28b3"}
Oct 30 22:45:03.394255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.393844    4581 scope.go:122] "RemoveContainer" containerID="8ff52e25dbf3cf134e311181026f9fd71a3ec77259c2b43e5b2fe2ad94ca28b3"
Oct 30 22:45:03.396370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.396349    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" event={"ID":"fd40171c-a1f5-484c-a078-ce72ef26cb90","Type":"ContainerStarted","Data":"c2cd8671b136c71d82c6c1322f3c47bdca58d1181cd8ddf4726e42e43c1720c2"}
Oct 30 22:45:03.399752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.399730    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" event={"ID":"e93290c9-59d9-48e6-990a-b18d680913ed","Type":"ContainerStarted","Data":"64c475a969ca7db4a763bbc99f73f1726b6612fb446204cb7d9ecf8c75b3ae97"}
Oct 30 22:45:03.400010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.399994    4581 scope.go:122] "RemoveContainer" containerID="64c475a969ca7db4a763bbc99f73f1726b6612fb446204cb7d9ecf8c75b3ae97"
Oct 30 22:45:03.405187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.404873    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" event={"ID":"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3","Type":"ContainerStarted","Data":"7c8705a16f48a86edd6166cceb527134628229137ef963c53f2276c8928a6836"}
Oct 30 22:45:03.406740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.406696    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" event={"ID":"abfeb292-6e71-4543-9215-a445f8226c13","Type":"ContainerStarted","Data":"7775b89f524f5c0a19a5a53ad80fed21c615de89f018b156fc8fb2671d3bf922"}
Oct 30 22:45:03.410661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.410029    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" event={"ID":"071439a8-bb97-4ab1-b4af-e0c88e7a585e","Type":"ContainerStarted","Data":"e92382b6eab784000013f1691249ae4ec7997ed3b4a8cbc2c6e83bfd4ccbba00"}
Oct 30 22:45:03.422299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.422256    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" podStartSLOduration=3.422242739 podStartE2EDuration="3.422242739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.389108087 +0000 UTC m=+2286.428046950" watchObservedRunningTime="2026-10-30 22:45:03.422242739 +0000 UTC m=+2287.461181622"
Oct 30 22:45:03.494946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.494887    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" podStartSLOduration=3.494873372 podStartE2EDuration="3.494873372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.455778919 +0000 UTC m=+2287.494717796" watchObservedRunningTime="2026-10-30 22:45:03.494873372 +0000 UTC m=+2287.533812258"
Oct 30 22:45:03.610473 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.610367    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" podStartSLOduration=3.610347914 podStartE2EDuration="3.610347914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.608444254 +0000 UTC m=+2287.647383154" watchObservedRunningTime="2026-10-30 22:45:03.610347914 +0000 UTC m=+2287.649286801"
Oct 30 22:45:03.645668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.645611    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" podStartSLOduration=3.645593432 podStartE2EDuration="3.645593432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.642422678 +0000 UTC m=+2287.681361563" watchObservedRunningTime="2026-10-30 22:45:03.645593432 +0000 UTC m=+2287.684532320"
Oct 30 22:45:03.728975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:03.728897    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" podStartSLOduration=3.728879291 podStartE2EDuration="3.728879291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.72583467 +0000 UTC m=+2287.764773556" watchObservedRunningTime="2026-10-30 22:45:03.728879291 +0000 UTC m=+2287.767818176"
Oct 30 22:45:04.422077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.422005    4581 generic.go:356] "Generic (PLEG): container finished" podID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerID="c2cd8671b136c71d82c6c1322f3c47bdca58d1181cd8ddf4726e42e43c1720c2" exitCode=60
Oct 30 22:45:04.422557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.422085    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" event={"ID":"fd40171c-a1f5-484c-a078-ce72ef26cb90","Type":"ContainerDied","Data":"c2cd8671b136c71d82c6c1322f3c47bdca58d1181cd8ddf4726e42e43c1720c2"}
Oct 30 22:45:04.422557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.422385    4581 scope.go:122] "RemoveContainer" containerID="c2cd8671b136c71d82c6c1322f3c47bdca58d1181cd8ddf4726e42e43c1720c2"
Oct 30 22:45:04.424004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.423977    4581 generic.go:356] "Generic (PLEG): container finished" podID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerID="3364a97915865ae1b375c21bc5559e57acef9ddb0243a4d64db8da424e801a3c" exitCode=60
Oct 30 22:45:04.424124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.424062    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" event={"ID":"34eb7cbf-d1bd-4141-b0a1-6d7833377c53","Type":"ContainerDied","Data":"3364a97915865ae1b375c21bc5559e57acef9ddb0243a4d64db8da424e801a3c"}
Oct 30 22:45:04.424319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.424298    4581 scope.go:122] "RemoveContainer" containerID="3364a97915865ae1b375c21bc5559e57acef9ddb0243a4d64db8da424e801a3c"
Oct 30 22:45:04.426196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.426169    4581 generic.go:356] "Generic (PLEG): container finished" podID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerID="c26f732c1acbd95ffe36686d2869423b3226a3bd9e22f81ac523b9588dcde979" exitCode=60
Oct 30 22:45:04.426344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.426241    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" event={"ID":"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3","Type":"ContainerDied","Data":"c26f732c1acbd95ffe36686d2869423b3226a3bd9e22f81ac523b9588dcde979"}
Oct 30 22:45:04.426555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.426498    4581 scope.go:122] "RemoveContainer" containerID="c26f732c1acbd95ffe36686d2869423b3226a3bd9e22f81ac523b9588dcde979"
Oct 30 22:45:04.431823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.431805    4581 generic.go:356] "Generic (PLEG): container finished" podID="e93290c9-59d9-48e6-990a-b18d680913ed" containerID="64c475a969ca7db4a763bbc99f73f1726b6612fb446204cb7d9ecf8c75b3ae97" exitCode=60
Oct 30 22:45:04.431938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.431876    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" event={"ID":"e93290c9-59d9-48e6-990a-b18d680913ed","Type":"ContainerDied","Data":"64c475a969ca7db4a763bbc99f73f1726b6612fb446204cb7d9ecf8c75b3ae97"}
Oct 30 22:45:04.433568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.433549    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" event={"ID":"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3","Type":"ContainerStarted","Data":"b72c3b698911103bc11819576b3bcfa3041b36da7fe59468adbf82e70af3f80b"}
Oct 30 22:45:04.435514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.435492    4581 generic.go:356] "Generic (PLEG): container finished" podID="169c00be-1d02-4314-83bd-d27450715d23" containerID="68374015f878603390297becd88e97c837566020b394e2c2ec8518cb97f3a331" exitCode=60
Oct 30 22:45:04.435616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.435581    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" event={"ID":"169c00be-1d02-4314-83bd-d27450715d23","Type":"ContainerDied","Data":"68374015f878603390297becd88e97c837566020b394e2c2ec8518cb97f3a331"}
Oct 30 22:45:04.435804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.435787    4581 scope.go:122] "RemoveContainer" containerID="68374015f878603390297becd88e97c837566020b394e2c2ec8518cb97f3a331"
Oct 30 22:45:04.439528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.439492    4581 generic.go:356] "Generic (PLEG): container finished" podID="abfeb292-6e71-4543-9215-a445f8226c13" containerID="7775b89f524f5c0a19a5a53ad80fed21c615de89f018b156fc8fb2671d3bf922" exitCode=60
Oct 30 22:45:04.439592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.439576    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" event={"ID":"abfeb292-6e71-4543-9215-a445f8226c13","Type":"ContainerDied","Data":"7775b89f524f5c0a19a5a53ad80fed21c615de89f018b156fc8fb2671d3bf922"}
Oct 30 22:45:04.439821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.439806    4581 scope.go:122] "RemoveContainer" containerID="7775b89f524f5c0a19a5a53ad80fed21c615de89f018b156fc8fb2671d3bf922"
Oct 30 22:45:04.441839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.441816    4581 generic.go:356] "Generic (PLEG): container finished" podID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerID="e92382b6eab784000013f1691249ae4ec7997ed3b4a8cbc2c6e83bfd4ccbba00" exitCode=60
Oct 30 22:45:04.441949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.441870    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" event={"ID":"071439a8-bb97-4ab1-b4af-e0c88e7a585e","Type":"ContainerDied","Data":"e92382b6eab784000013f1691249ae4ec7997ed3b4a8cbc2c6e83bfd4ccbba00"}
Oct 30 22:45:04.442095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.442079    4581 scope.go:122] "RemoveContainer" containerID="e92382b6eab784000013f1691249ae4ec7997ed3b4a8cbc2c6e83bfd4ccbba00"
Oct 30 22:45:04.443801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.443773    4581 generic.go:356] "Generic (PLEG): container finished" podID="31f18846-74ba-48c9-a797-0f33d1820e16" containerID="f5c881c0a6e060516c23117a77fbb90110328257e5612b4306d1f2436f7dfccf" exitCode=60
Oct 30 22:45:04.443903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.443797    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" event={"ID":"31f18846-74ba-48c9-a797-0f33d1820e16","Type":"ContainerDied","Data":"f5c881c0a6e060516c23117a77fbb90110328257e5612b4306d1f2436f7dfccf"}
Oct 30 22:45:04.444051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.444036    4581 scope.go:122] "RemoveContainer" containerID="f5c881c0a6e060516c23117a77fbb90110328257e5612b4306d1f2436f7dfccf"
Oct 30 22:45:04.445676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.445652    4581 generic.go:356] "Generic (PLEG): container finished" podID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerID="2962e9f3181496683622ef0d26f69b33a342d0df64e1ccaa77662f5298251ab3" exitCode=60
Oct 30 22:45:04.445858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.445766    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" event={"ID":"dd3f411a-89df-4177-bb36-b9bdcca5e8c2","Type":"ContainerDied","Data":"2962e9f3181496683622ef0d26f69b33a342d0df64e1ccaa77662f5298251ab3"}
Oct 30 22:45:04.445966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.445948    4581 scope.go:122] "RemoveContainer" containerID="2962e9f3181496683622ef0d26f69b33a342d0df64e1ccaa77662f5298251ab3"
Oct 30 22:45:04.451888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.450562    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" event={"ID":"a5f7488c-2538-4dc6-818b-306e295933cc","Type":"ContainerStarted","Data":"a88a46e8c77268a102b17e437b0ce5ba3b88d44aeb67c3a4b529c359a7a01913"}
Oct 30 22:45:04.454242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.454199    4581 generic.go:356] "Generic (PLEG): container finished" podID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerID="94418a579db6b0284efff7485f3d07a3ee111e70093d53dbec7eba545e126cdd" exitCode=60
Oct 30 22:45:04.454434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.454302    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" event={"ID":"fb334b3a-3f29-4505-ae06-d4ce0917bebc","Type":"ContainerDied","Data":"94418a579db6b0284efff7485f3d07a3ee111e70093d53dbec7eba545e126cdd"}
Oct 30 22:45:04.455059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.455043    4581 scope.go:122] "RemoveContainer" containerID="94418a579db6b0284efff7485f3d07a3ee111e70093d53dbec7eba545e126cdd"
Oct 30 22:45:04.456999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.456980    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" event={"ID":"abeab3fd-da86-4674-943d-0560abd2e752","Type":"ContainerStarted","Data":"ef37f336c205e7f308802440f607b593d849c2063fae498d7cc88c3ee1cf1599"}
Oct 30 22:45:04.473430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.473359    4581 generic.go:356] "Generic (PLEG): container finished" podID="23d0420e-107e-40c7-8d92-85c50abb696d" containerID="148e0a493a0da0fc06704b7324aee18dc63ca0753ab6dbbc9b9f216bc9211fc8" exitCode=60
Oct 30 22:45:04.475238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.473807    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" event={"ID":"23d0420e-107e-40c7-8d92-85c50abb696d","Type":"ContainerDied","Data":"148e0a493a0da0fc06704b7324aee18dc63ca0753ab6dbbc9b9f216bc9211fc8"}
Oct 30 22:45:04.476322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.475412    4581 scope.go:122] "RemoveContainer" containerID="148e0a493a0da0fc06704b7324aee18dc63ca0753ab6dbbc9b9f216bc9211fc8"
Oct 30 22:45:04.602303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.602275    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" event={"ID":"decc6e06-9261-4213-a87b-26fb98bcbe50","Type":"ContainerStarted","Data":"9b4119d0378670ae87f4ed75b7f96ca0f5a70ddade8d9273579a3602f5bbc9f9"}
Oct 30 22:45:04.607423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.607392    4581 generic.go:356] "Generic (PLEG): container finished" podID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerID="75fcf17f9873c773e5176bf73b39d5534435d2ca914a35f4ddf2cf3f5638f0f7" exitCode=60
Oct 30 22:45:04.608044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.607978    4581 scope.go:122] "RemoveContainer" containerID="75fcf17f9873c773e5176bf73b39d5534435d2ca914a35f4ddf2cf3f5638f0f7"
Oct 30 22:45:04.608757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.607996    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" event={"ID":"2296d534-4a85-4b40-a54e-c8b2a67e2aa2","Type":"ContainerDied","Data":"75fcf17f9873c773e5176bf73b39d5534435d2ca914a35f4ddf2cf3f5638f0f7"}
Oct 30 22:45:04.727202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.727138    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" podStartSLOduration=4.727118518 podStartE2EDuration="4.727118518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.720932167 +0000 UTC m=+2288.759871054" watchObservedRunningTime="2026-10-30 22:45:04.727118518 +0000 UTC m=+2288.766057406"
Oct 30 22:45:04.868965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.868910    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" podStartSLOduration=4.868891507 podStartE2EDuration="4.868891507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.860471082 +0000 UTC m=+2288.899409969" watchObservedRunningTime="2026-10-30 22:45:04.868891507 +0000 UTC m=+2288.907830394"
Oct 30 22:45:04.942760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:04.942699    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" podStartSLOduration=4.94268369 podStartE2EDuration="4.94268369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.936863469 +0000 UTC m=+2288.975802366" watchObservedRunningTime="2026-10-30 22:45:04.94268369 +0000 UTC m=+2288.981622595"
Oct 30 22:45:05.034249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.025128    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" podStartSLOduration=5.025103399 podStartE2EDuration="5.025103399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.005592663 +0000 UTC m=+2289.044531553" watchObservedRunningTime="2026-10-30 22:45:05.025103399 +0000 UTC m=+2289.064042286"
Oct 30 22:45:05.620448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.620415    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" event={"ID":"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3","Type":"ContainerStarted","Data":"ba428f65f03f84d6fafa15beac1b29f5dfd2a088250c1532b561f7a344944250"}
Oct 30 22:45:05.622966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.622939    4581 generic.go:356] "Generic (PLEG): container finished" podID="e93290c9-59d9-48e6-990a-b18d680913ed" containerID="bade1104857a54c105235c6a4bf52fa2a1006b78dc1879728c0107a56b37b7c3" exitCode=60
Oct 30 22:45:05.623106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.623027    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" event={"ID":"e93290c9-59d9-48e6-990a-b18d680913ed","Type":"ContainerDied","Data":"bade1104857a54c105235c6a4bf52fa2a1006b78dc1879728c0107a56b37b7c3"}
Oct 30 22:45:05.623106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.623089    4581 scope.go:122] "RemoveContainer" containerID="64c475a969ca7db4a763bbc99f73f1726b6612fb446204cb7d9ecf8c75b3ae97"
Oct 30 22:45:05.625263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.623436    4581 scope.go:122] "RemoveContainer" containerID="bade1104857a54c105235c6a4bf52fa2a1006b78dc1879728c0107a56b37b7c3"
Oct 30 22:45:05.625263 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:05.623808    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq_odh-ai-gateway-infra(e93290c9-59d9-48e6-990a-b18d680913ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" podUID="e93290c9-59d9-48e6-990a-b18d680913ed"
Oct 30 22:45:05.626727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.626704    4581 generic.go:356] "Generic (PLEG): container finished" podID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerID="b72c3b698911103bc11819576b3bcfa3041b36da7fe59468adbf82e70af3f80b" exitCode=60
Oct 30 22:45:05.626832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.626775    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" event={"ID":"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3","Type":"ContainerDied","Data":"b72c3b698911103bc11819576b3bcfa3041b36da7fe59468adbf82e70af3f80b"}
Oct 30 22:45:05.626998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.626984    4581 scope.go:122] "RemoveContainer" containerID="b72c3b698911103bc11819576b3bcfa3041b36da7fe59468adbf82e70af3f80b"
Oct 30 22:45:05.629097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.629073    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" event={"ID":"169c00be-1d02-4314-83bd-d27450715d23","Type":"ContainerStarted","Data":"23e159785f52ff901070c28f0ea97489d7074fe92b843e093b1e550bc35d80de"}
Oct 30 22:45:05.631438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.631417    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" event={"ID":"abfeb292-6e71-4543-9215-a445f8226c13","Type":"ContainerStarted","Data":"6973e1f5217f60a518eeaa63d5de072d9c76776dd4f270229018ca759a79b4d2"}
Oct 30 22:45:05.633706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.633658    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" event={"ID":"071439a8-bb97-4ab1-b4af-e0c88e7a585e","Type":"ContainerStarted","Data":"5c11f93624efc5a8546ba61d8bfbe2277e1e182070beb541d63af2ff6a6c23f4"}
Oct 30 22:45:05.636057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.636033    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" event={"ID":"31f18846-74ba-48c9-a797-0f33d1820e16","Type":"ContainerStarted","Data":"467bd75710cac9009c50af7c1f64cc2e214732f7e15fe78e6f63d34d6241bdca"}
Oct 30 22:45:05.638298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.638275    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" event={"ID":"dd3f411a-89df-4177-bb36-b9bdcca5e8c2","Type":"ContainerStarted","Data":"fb8d3ae0654b0cbbbcc9a2b2fa2dbf54a754609d87d2965c8b9db797fcbc37e7"}
Oct 30 22:45:05.640330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.640312    4581 generic.go:356] "Generic (PLEG): container finished" podID="a5f7488c-2538-4dc6-818b-306e295933cc" containerID="a88a46e8c77268a102b17e437b0ce5ba3b88d44aeb67c3a4b529c359a7a01913" exitCode=60
Oct 30 22:45:05.640433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.640399    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" event={"ID":"a5f7488c-2538-4dc6-818b-306e295933cc","Type":"ContainerDied","Data":"a88a46e8c77268a102b17e437b0ce5ba3b88d44aeb67c3a4b529c359a7a01913"}
Oct 30 22:45:05.640595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.640577    4581 scope.go:122] "RemoveContainer" containerID="a88a46e8c77268a102b17e437b0ce5ba3b88d44aeb67c3a4b529c359a7a01913"
Oct 30 22:45:05.640861 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:05.640840    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw_odh-ai-gateway-infra(a5f7488c-2538-4dc6-818b-306e295933cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" podUID="a5f7488c-2538-4dc6-818b-306e295933cc"
Oct 30 22:45:05.662249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.642684    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" event={"ID":"fb334b3a-3f29-4505-ae06-d4ce0917bebc","Type":"ContainerStarted","Data":"e0494094f2ce291b277e2cbf7bcf545dac115b0ab5bee86a938f5e0214d4b543"}
Oct 30 22:45:05.662249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.645839    4581 generic.go:356] "Generic (PLEG): container finished" podID="abeab3fd-da86-4674-943d-0560abd2e752" containerID="ef37f336c205e7f308802440f607b593d849c2063fae498d7cc88c3ee1cf1599" exitCode=60
Oct 30 22:45:05.662249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.645910    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" event={"ID":"abeab3fd-da86-4674-943d-0560abd2e752","Type":"ContainerDied","Data":"ef37f336c205e7f308802440f607b593d849c2063fae498d7cc88c3ee1cf1599"}
Oct 30 22:45:05.662249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.646296    4581 scope.go:122] "RemoveContainer" containerID="ef37f336c205e7f308802440f607b593d849c2063fae498d7cc88c3ee1cf1599"
Oct 30 22:45:05.662249 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:05.646604    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz_odh-ai-gateway-infra(abeab3fd-da86-4674-943d-0560abd2e752)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" podUID="abeab3fd-da86-4674-943d-0560abd2e752"
Oct 30 22:45:05.667292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.667039    4581 generic.go:356] "Generic (PLEG): container finished" podID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerID="52ee624532f686244f779cbd0262edcf521de8d6f77954778a8321fd146e26d7" exitCode=60
Oct 30 22:45:05.667292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.667273    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" event={"ID":"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb","Type":"ContainerDied","Data":"52ee624532f686244f779cbd0262edcf521de8d6f77954778a8321fd146e26d7"}
Oct 30 22:45:05.667965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.667905    4581 scope.go:122] "RemoveContainer" containerID="52ee624532f686244f779cbd0262edcf521de8d6f77954778a8321fd146e26d7"
Oct 30 22:45:05.668652 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:05.668405    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc_odh-ai-gateway-infra(973aa012-ec8e-455f-9c5a-4fd55e3d4dbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb"
Oct 30 22:45:05.673233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.671667    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" podStartSLOduration=5.671654176 podStartE2EDuration="5.671654176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.662511463 +0000 UTC m=+2289.701450353" watchObservedRunningTime="2026-10-30 22:45:05.671654176 +0000 UTC m=+2289.710593087"
Oct 30 22:45:05.673233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.672856    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" event={"ID":"23d0420e-107e-40c7-8d92-85c50abb696d","Type":"ContainerStarted","Data":"8fa4ff1bbc4e229936105172eb0193e7ea8abfd9d09c0d5e38bf260be2619ed7"}
Oct 30 22:45:05.677027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.676992    4581 generic.go:356] "Generic (PLEG): container finished" podID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerID="9b4119d0378670ae87f4ed75b7f96ca0f5a70ddade8d9273579a3602f5bbc9f9" exitCode=60
Oct 30 22:45:05.677159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.677101    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" event={"ID":"decc6e06-9261-4213-a87b-26fb98bcbe50","Type":"ContainerDied","Data":"9b4119d0378670ae87f4ed75b7f96ca0f5a70ddade8d9273579a3602f5bbc9f9"}
Oct 30 22:45:05.677551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.677532    4581 scope.go:122] "RemoveContainer" containerID="9b4119d0378670ae87f4ed75b7f96ca0f5a70ddade8d9273579a3602f5bbc9f9"
Oct 30 22:45:05.677986 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:05.677964    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv_odh-ai-gateway-infra(decc6e06-9261-4213-a87b-26fb98bcbe50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50"
Oct 30 22:45:05.680994 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.680908    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" event={"ID":"2296d534-4a85-4b40-a54e-c8b2a67e2aa2","Type":"ContainerStarted","Data":"ceb5bd30e0074f63c4459f090513b9912df7f1ddcb0599944232407a87d09378"}
Oct 30 22:45:05.683459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.683433    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" event={"ID":"fd40171c-a1f5-484c-a078-ce72ef26cb90","Type":"ContainerStarted","Data":"049bb9121132cf5ab3a110fc7ebcb43fbfa6f2a6c565f672dbc1e099d80a3f2a"}
Oct 30 22:45:05.688284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.686543    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" event={"ID":"34eb7cbf-d1bd-4141-b0a1-6d7833377c53","Type":"ContainerStarted","Data":"dc70f0c1e62acf0910e9b62db72f5e1a599446c2b4116fb47f88bb1c8aefcd2c"}
Oct 30 22:45:05.740569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.739609    4581 scope.go:122] "RemoveContainer" containerID="952d949f5475809f88ac09eb9b08455e6c856add8e2c987db3804c0e617864e6"
Oct 30 22:45:05.770848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.770790    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" podStartSLOduration=5.770772197 podStartE2EDuration="5.770772197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.766996043 +0000 UTC m=+2289.805934931" watchObservedRunningTime="2026-10-30 22:45:05.770772197 +0000 UTC m=+2289.809711085"
Oct 30 22:45:05.771271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.771209    4581 scope.go:122] "RemoveContainer" containerID="1c6ee7b7d3e182d67935d8c6d97aec2cbf3da0c50c1e5a8b0deacde3d333c3d0"
Oct 30 22:45:05.790706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.790684    4581 scope.go:122] "RemoveContainer" containerID="bf75fe4bc38224d477f5a48c645fc7586f571273e216768bd9025a0da4b7cf29"
Oct 30 22:45:05.807302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.807278    4581 scope.go:122] "RemoveContainer" containerID="8ff52e25dbf3cf134e311181026f9fd71a3ec77259c2b43e5b2fe2ad94ca28b3"
Oct 30 22:45:05.840410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.840358    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" podStartSLOduration=5.840344932 podStartE2EDuration="5.840344932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.835309655 +0000 UTC m=+2289.874248544" watchObservedRunningTime="2026-10-30 22:45:05.840344932 +0000 UTC m=+2289.879283817"
Oct 30 22:45:05.873796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:05.873736    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" podStartSLOduration=5.873717005 podStartE2EDuration="5.873717005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.868520632 +0000 UTC m=+2289.907459519" watchObservedRunningTime="2026-10-30 22:45:05.873717005 +0000 UTC m=+2289.912655891"
Oct 30 22:45:06.002287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.002231    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" podStartSLOduration=6.002197891 podStartE2EDuration="6.002197891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:06.001258622 +0000 UTC m=+2290.040197510" watchObservedRunningTime="2026-10-30 22:45:06.002197891 +0000 UTC m=+2290.041136778"
Oct 30 22:45:06.076910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.076803    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" podStartSLOduration=6.076786567 podStartE2EDuration="6.076786567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:06.076068342 +0000 UTC m=+2290.115007230" watchObservedRunningTime="2026-10-30 22:45:06.076786567 +0000 UTC m=+2290.115725452"
Oct 30 22:45:06.187870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.187805    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" podStartSLOduration=6.187784286 podStartE2EDuration="6.187784286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:06.184774553 +0000 UTC m=+2290.223713440" watchObservedRunningTime="2026-10-30 22:45:06.187784286 +0000 UTC m=+2290.226723172"
Oct 30 22:45:06.703661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.703563    4581 generic.go:356] "Generic (PLEG): container finished" podID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerID="ba428f65f03f84d6fafa15beac1b29f5dfd2a088250c1532b561f7a344944250" exitCode=60
Oct 30 22:45:06.704068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.703830    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" event={"ID":"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3","Type":"ContainerDied","Data":"ba428f65f03f84d6fafa15beac1b29f5dfd2a088250c1532b561f7a344944250"}
Oct 30 22:45:06.704068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.703898    4581 scope.go:122] "RemoveContainer" containerID="c26f732c1acbd95ffe36686d2869423b3226a3bd9e22f81ac523b9588dcde979"
Oct 30 22:45:06.704149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.704084    4581 scope.go:122] "RemoveContainer" containerID="ba428f65f03f84d6fafa15beac1b29f5dfd2a088250c1532b561f7a344944250"
Oct 30 22:45:06.704615 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.704587    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5_odh-ai-gateway-infra(8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3"
Oct 30 22:45:06.706991 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.706684    4581 scope.go:122] "RemoveContainer" containerID="bade1104857a54c105235c6a4bf52fa2a1006b78dc1879728c0107a56b37b7c3"
Oct 30 22:45:06.707098 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.707048    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq_odh-ai-gateway-infra(e93290c9-59d9-48e6-990a-b18d680913ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" podUID="e93290c9-59d9-48e6-990a-b18d680913ed"
Oct 30 22:45:06.708740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.708717    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" event={"ID":"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3","Type":"ContainerStarted","Data":"8cb3f87c9d971f02d6822d4daeb4e90eb436a426d48116b6a1052e1f1115adc6"}
Oct 30 22:45:06.713072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.713048    4581 generic.go:356] "Generic (PLEG): container finished" podID="169c00be-1d02-4314-83bd-d27450715d23" containerID="23e159785f52ff901070c28f0ea97489d7074fe92b843e093b1e550bc35d80de" exitCode=60
Oct 30 22:45:06.713194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.713142    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" event={"ID":"169c00be-1d02-4314-83bd-d27450715d23","Type":"ContainerDied","Data":"23e159785f52ff901070c28f0ea97489d7074fe92b843e093b1e550bc35d80de"}
Oct 30 22:45:06.713426 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.713397    4581 scope.go:122] "RemoveContainer" containerID="23e159785f52ff901070c28f0ea97489d7074fe92b843e093b1e550bc35d80de"
Oct 30 22:45:06.713840 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.713811    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm_odh-ai-gateway-infra(169c00be-1d02-4314-83bd-d27450715d23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" podUID="169c00be-1d02-4314-83bd-d27450715d23"
Oct 30 22:45:06.715358 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.715336    4581 generic.go:356] "Generic (PLEG): container finished" podID="abfeb292-6e71-4543-9215-a445f8226c13" containerID="6973e1f5217f60a518eeaa63d5de072d9c76776dd4f270229018ca759a79b4d2" exitCode=60
Oct 30 22:45:06.715470 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.715420    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" event={"ID":"abfeb292-6e71-4543-9215-a445f8226c13","Type":"ContainerDied","Data":"6973e1f5217f60a518eeaa63d5de072d9c76776dd4f270229018ca759a79b4d2"}
Oct 30 22:45:06.715672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.715653    4581 scope.go:122] "RemoveContainer" containerID="6973e1f5217f60a518eeaa63d5de072d9c76776dd4f270229018ca759a79b4d2"
Oct 30 22:45:06.716035 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.716009    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b_odh-ai-gateway-infra(abfeb292-6e71-4543-9215-a445f8226c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" podUID="abfeb292-6e71-4543-9215-a445f8226c13"
Oct 30 22:45:06.717703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.717678    4581 generic.go:356] "Generic (PLEG): container finished" podID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerID="5c11f93624efc5a8546ba61d8bfbe2277e1e182070beb541d63af2ff6a6c23f4" exitCode=60
Oct 30 22:45:06.717790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.717759    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" event={"ID":"071439a8-bb97-4ab1-b4af-e0c88e7a585e","Type":"ContainerDied","Data":"5c11f93624efc5a8546ba61d8bfbe2277e1e182070beb541d63af2ff6a6c23f4"}
Oct 30 22:45:06.718023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.718007    4581 scope.go:122] "RemoveContainer" containerID="5c11f93624efc5a8546ba61d8bfbe2277e1e182070beb541d63af2ff6a6c23f4"
Oct 30 22:45:06.718385 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.718361    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc_odh-ai-gateway-infra(071439a8-bb97-4ab1-b4af-e0c88e7a585e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e"
Oct 30 22:45:06.719934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.719912    4581 generic.go:356] "Generic (PLEG): container finished" podID="31f18846-74ba-48c9-a797-0f33d1820e16" containerID="467bd75710cac9009c50af7c1f64cc2e214732f7e15fe78e6f63d34d6241bdca" exitCode=60
Oct 30 22:45:06.720011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.719984    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" event={"ID":"31f18846-74ba-48c9-a797-0f33d1820e16","Type":"ContainerDied","Data":"467bd75710cac9009c50af7c1f64cc2e214732f7e15fe78e6f63d34d6241bdca"}
Oct 30 22:45:06.720194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.720179    4581 scope.go:122] "RemoveContainer" containerID="467bd75710cac9009c50af7c1f64cc2e214732f7e15fe78e6f63d34d6241bdca"
Oct 30 22:45:06.720554 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.720527    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8_odh-ai-gateway-infra(31f18846-74ba-48c9-a797-0f33d1820e16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" podUID="31f18846-74ba-48c9-a797-0f33d1820e16"
Oct 30 22:45:06.722512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.722491    4581 generic.go:356] "Generic (PLEG): container finished" podID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerID="fb8d3ae0654b0cbbbcc9a2b2fa2dbf54a754609d87d2965c8b9db797fcbc37e7" exitCode=60
Oct 30 22:45:06.722603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.722564    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" event={"ID":"dd3f411a-89df-4177-bb36-b9bdcca5e8c2","Type":"ContainerDied","Data":"fb8d3ae0654b0cbbbcc9a2b2fa2dbf54a754609d87d2965c8b9db797fcbc37e7"}
Oct 30 22:45:06.722799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.722767    4581 scope.go:122] "RemoveContainer" containerID="fb8d3ae0654b0cbbbcc9a2b2fa2dbf54a754609d87d2965c8b9db797fcbc37e7"
Oct 30 22:45:06.722939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.722922    4581 scope.go:122] "RemoveContainer" containerID="68374015f878603390297becd88e97c837566020b394e2c2ec8518cb97f3a331"
Oct 30 22:45:06.723054 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.723033    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v_odh-ai-gateway-infra(dd3f411a-89df-4177-bb36-b9bdcca5e8c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2"
Oct 30 22:45:06.724593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.724573    4581 scope.go:122] "RemoveContainer" containerID="a88a46e8c77268a102b17e437b0ce5ba3b88d44aeb67c3a4b529c359a7a01913"
Oct 30 22:45:06.724827 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.724806    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw_odh-ai-gateway-infra(a5f7488c-2538-4dc6-818b-306e295933cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" podUID="a5f7488c-2538-4dc6-818b-306e295933cc"
Oct 30 22:45:06.726405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.726388    4581 generic.go:356] "Generic (PLEG): container finished" podID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerID="e0494094f2ce291b277e2cbf7bcf545dac115b0ab5bee86a938f5e0214d4b543" exitCode=60
Oct 30 22:45:06.726487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.726449    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" event={"ID":"fb334b3a-3f29-4505-ae06-d4ce0917bebc","Type":"ContainerDied","Data":"e0494094f2ce291b277e2cbf7bcf545dac115b0ab5bee86a938f5e0214d4b543"}
Oct 30 22:45:06.726655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.726643    4581 scope.go:122] "RemoveContainer" containerID="e0494094f2ce291b277e2cbf7bcf545dac115b0ab5bee86a938f5e0214d4b543"
Oct 30 22:45:06.726916 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.726896    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd_odh-ai-gateway-infra(fb334b3a-3f29-4505-ae06-d4ce0917bebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc"
Oct 30 22:45:06.728745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.728728    4581 scope.go:122] "RemoveContainer" containerID="ef37f336c205e7f308802440f607b593d849c2063fae498d7cc88c3ee1cf1599"
Oct 30 22:45:06.728948 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.728931    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz_odh-ai-gateway-infra(abeab3fd-da86-4674-943d-0560abd2e752)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" podUID="abeab3fd-da86-4674-943d-0560abd2e752"
Oct 30 22:45:06.730663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.730644    4581 scope.go:122] "RemoveContainer" containerID="52ee624532f686244f779cbd0262edcf521de8d6f77954778a8321fd146e26d7"
Oct 30 22:45:06.730903 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.730879    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc_odh-ai-gateway-infra(973aa012-ec8e-455f-9c5a-4fd55e3d4dbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb"
Oct 30 22:45:06.732485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.732464    4581 generic.go:356] "Generic (PLEG): container finished" podID="23d0420e-107e-40c7-8d92-85c50abb696d" containerID="8fa4ff1bbc4e229936105172eb0193e7ea8abfd9d09c0d5e38bf260be2619ed7" exitCode=60
Oct 30 22:45:06.732603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.732548    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" event={"ID":"23d0420e-107e-40c7-8d92-85c50abb696d","Type":"ContainerDied","Data":"8fa4ff1bbc4e229936105172eb0193e7ea8abfd9d09c0d5e38bf260be2619ed7"}
Oct 30 22:45:06.732979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.732960    4581 scope.go:122] "RemoveContainer" containerID="8fa4ff1bbc4e229936105172eb0193e7ea8abfd9d09c0d5e38bf260be2619ed7"
Oct 30 22:45:06.733258 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.733236    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw_odh-ai-gateway-infra(23d0420e-107e-40c7-8d92-85c50abb696d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" podUID="23d0420e-107e-40c7-8d92-85c50abb696d"
Oct 30 22:45:06.734366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.734350    4581 scope.go:122] "RemoveContainer" containerID="9b4119d0378670ae87f4ed75b7f96ca0f5a70ddade8d9273579a3602f5bbc9f9"
Oct 30 22:45:06.734599 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.734578    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv_odh-ai-gateway-infra(decc6e06-9261-4213-a87b-26fb98bcbe50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50"
Oct 30 22:45:06.736252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.736228    4581 generic.go:356] "Generic (PLEG): container finished" podID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerID="ceb5bd30e0074f63c4459f090513b9912df7f1ddcb0599944232407a87d09378" exitCode=60
Oct 30 22:45:06.736343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.736257    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" event={"ID":"2296d534-4a85-4b40-a54e-c8b2a67e2aa2","Type":"ContainerDied","Data":"ceb5bd30e0074f63c4459f090513b9912df7f1ddcb0599944232407a87d09378"}
Oct 30 22:45:06.736513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.736499    4581 scope.go:122] "RemoveContainer" containerID="ceb5bd30e0074f63c4459f090513b9912df7f1ddcb0599944232407a87d09378"
Oct 30 22:45:06.736741 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.736713    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85_odh-ai-gateway-infra(2296d534-4a85-4b40-a54e-c8b2a67e2aa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2"
Oct 30 22:45:06.740324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.740305    4581 scope.go:122] "RemoveContainer" containerID="7775b89f524f5c0a19a5a53ad80fed21c615de89f018b156fc8fb2671d3bf922"
Oct 30 22:45:06.750205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.750180    4581 generic.go:356] "Generic (PLEG): container finished" podID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerID="049bb9121132cf5ab3a110fc7ebcb43fbfa6f2a6c565f672dbc1e099d80a3f2a" exitCode=60
Oct 30 22:45:06.750424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.750399    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" event={"ID":"fd40171c-a1f5-484c-a078-ce72ef26cb90","Type":"ContainerDied","Data":"049bb9121132cf5ab3a110fc7ebcb43fbfa6f2a6c565f672dbc1e099d80a3f2a"}
Oct 30 22:45:06.750767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.750748    4581 scope.go:122] "RemoveContainer" containerID="049bb9121132cf5ab3a110fc7ebcb43fbfa6f2a6c565f672dbc1e099d80a3f2a"
Oct 30 22:45:06.751058 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.751033    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d_odh-ai-gateway-infra(fd40171c-a1f5-484c-a078-ce72ef26cb90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90"
Oct 30 22:45:06.754115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.754034    4581 generic.go:356] "Generic (PLEG): container finished" podID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerID="dc70f0c1e62acf0910e9b62db72f5e1a599446c2b4116fb47f88bb1c8aefcd2c" exitCode=60
Oct 30 22:45:06.754115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.754103    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" event={"ID":"34eb7cbf-d1bd-4141-b0a1-6d7833377c53","Type":"ContainerDied","Data":"dc70f0c1e62acf0910e9b62db72f5e1a599446c2b4116fb47f88bb1c8aefcd2c"}
Oct 30 22:45:06.754424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.754405    4581 scope.go:122] "RemoveContainer" containerID="dc70f0c1e62acf0910e9b62db72f5e1a599446c2b4116fb47f88bb1c8aefcd2c"
Oct 30 22:45:06.754847 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:06.754765    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr_odh-ai-gateway-infra(34eb7cbf-d1bd-4141-b0a1-6d7833377c53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53"
Oct 30 22:45:06.768581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.768557    4581 scope.go:122] "RemoveContainer" containerID="e92382b6eab784000013f1691249ae4ec7997ed3b4a8cbc2c6e83bfd4ccbba00"
Oct 30 22:45:06.786729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.786702    4581 scope.go:122] "RemoveContainer" containerID="f5c881c0a6e060516c23117a77fbb90110328257e5612b4306d1f2436f7dfccf"
Oct 30 22:45:06.802932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.802903    4581 scope.go:122] "RemoveContainer" containerID="2962e9f3181496683622ef0d26f69b33a342d0df64e1ccaa77662f5298251ab3"
Oct 30 22:45:06.896769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.896654    4581 scope.go:122] "RemoveContainer" containerID="94418a579db6b0284efff7485f3d07a3ee111e70093d53dbec7eba545e126cdd"
Oct 30 22:45:06.935079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.935056    4581 scope.go:122] "RemoveContainer" containerID="148e0a493a0da0fc06704b7324aee18dc63ca0753ab6dbbc9b9f216bc9211fc8"
Oct 30 22:45:06.950153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.950128    4581 scope.go:122] "RemoveContainer" containerID="75fcf17f9873c773e5176bf73b39d5534435d2ca914a35f4ddf2cf3f5638f0f7"
Oct 30 22:45:06.969046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.968554    4581 scope.go:122] "RemoveContainer" containerID="c2cd8671b136c71d82c6c1322f3c47bdca58d1181cd8ddf4726e42e43c1720c2"
Oct 30 22:45:06.984497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:06.984462    4581 scope.go:122] "RemoveContainer" containerID="3364a97915865ae1b375c21bc5559e57acef9ddb0243a4d64db8da424e801a3c"
Oct 30 22:45:07.767405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.767374    4581 scope.go:122] "RemoveContainer" containerID="6973e1f5217f60a518eeaa63d5de072d9c76776dd4f270229018ca759a79b4d2"
Oct 30 22:45:07.768365 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.767847    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b_odh-ai-gateway-infra(abfeb292-6e71-4543-9215-a445f8226c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" podUID="abfeb292-6e71-4543-9215-a445f8226c13"
Oct 30 22:45:07.769446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.769425    4581 scope.go:122] "RemoveContainer" containerID="5c11f93624efc5a8546ba61d8bfbe2277e1e182070beb541d63af2ff6a6c23f4"
Oct 30 22:45:07.769808 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.769787    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc_odh-ai-gateway-infra(071439a8-bb97-4ab1-b4af-e0c88e7a585e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e"
Oct 30 22:45:07.771435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.771414    4581 scope.go:122] "RemoveContainer" containerID="467bd75710cac9009c50af7c1f64cc2e214732f7e15fe78e6f63d34d6241bdca"
Oct 30 22:45:07.771750 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.771725    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8_odh-ai-gateway-infra(31f18846-74ba-48c9-a797-0f33d1820e16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" podUID="31f18846-74ba-48c9-a797-0f33d1820e16"
Oct 30 22:45:07.773061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.773043    4581 scope.go:122] "RemoveContainer" containerID="fb8d3ae0654b0cbbbcc9a2b2fa2dbf54a754609d87d2965c8b9db797fcbc37e7"
Oct 30 22:45:07.773367 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.773343    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v_odh-ai-gateway-infra(dd3f411a-89df-4177-bb36-b9bdcca5e8c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2"
Oct 30 22:45:07.774865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.774847    4581 scope.go:122] "RemoveContainer" containerID="e0494094f2ce291b277e2cbf7bcf545dac115b0ab5bee86a938f5e0214d4b543"
Oct 30 22:45:07.775135 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.775108    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd_odh-ai-gateway-infra(fb334b3a-3f29-4505-ae06-d4ce0917bebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc"
Oct 30 22:45:07.776760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.776744    4581 scope.go:122] "RemoveContainer" containerID="8fa4ff1bbc4e229936105172eb0193e7ea8abfd9d09c0d5e38bf260be2619ed7"
Oct 30 22:45:07.776991 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.776969    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw_odh-ai-gateway-infra(23d0420e-107e-40c7-8d92-85c50abb696d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" podUID="23d0420e-107e-40c7-8d92-85c50abb696d"
Oct 30 22:45:07.778577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.778562    4581 scope.go:122] "RemoveContainer" containerID="ceb5bd30e0074f63c4459f090513b9912df7f1ddcb0599944232407a87d09378"
Oct 30 22:45:07.778843 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.778819    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85_odh-ai-gateway-infra(2296d534-4a85-4b40-a54e-c8b2a67e2aa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2"
Oct 30 22:45:07.780535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.780513    4581 scope.go:122] "RemoveContainer" containerID="049bb9121132cf5ab3a110fc7ebcb43fbfa6f2a6c565f672dbc1e099d80a3f2a"
Oct 30 22:45:07.780751 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.780732    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d_odh-ai-gateway-infra(fd40171c-a1f5-484c-a078-ce72ef26cb90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90"
Oct 30 22:45:07.782372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.782352    4581 scope.go:122] "RemoveContainer" containerID="dc70f0c1e62acf0910e9b62db72f5e1a599446c2b4116fb47f88bb1c8aefcd2c"
Oct 30 22:45:07.782631 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.782606    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr_odh-ai-gateway-infra(34eb7cbf-d1bd-4141-b0a1-6d7833377c53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53"
Oct 30 22:45:07.784141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.784124    4581 scope.go:122] "RemoveContainer" containerID="ba428f65f03f84d6fafa15beac1b29f5dfd2a088250c1532b561f7a344944250"
Oct 30 22:45:07.784399 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.784376    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5_odh-ai-gateway-infra(8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3"
Oct 30 22:45:07.786182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.786158    4581 generic.go:356] "Generic (PLEG): container finished" podID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerID="8cb3f87c9d971f02d6822d4daeb4e90eb436a426d48116b6a1052e1f1115adc6" exitCode=60
Oct 30 22:45:07.786313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.786248    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" event={"ID":"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3","Type":"ContainerDied","Data":"8cb3f87c9d971f02d6822d4daeb4e90eb436a426d48116b6a1052e1f1115adc6"}
Oct 30 22:45:07.786313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.786300    4581 scope.go:122] "RemoveContainer" containerID="b72c3b698911103bc11819576b3bcfa3041b36da7fe59468adbf82e70af3f80b"
Oct 30 22:45:07.786486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.786464    4581 scope.go:122] "RemoveContainer" containerID="8cb3f87c9d971f02d6822d4daeb4e90eb436a426d48116b6a1052e1f1115adc6"
Oct 30 22:45:07.786736 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.786715    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw_odh-ai-gateway-infra(71fc3c7a-b8bd-4e23-a857-09ffb342b6e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3"
Oct 30 22:45:07.788518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:07.788499    4581 scope.go:122] "RemoveContainer" containerID="23e159785f52ff901070c28f0ea97489d7074fe92b843e093b1e550bc35d80de"
Oct 30 22:45:07.788748 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:07.788726    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm_odh-ai-gateway-infra(169c00be-1d02-4314-83bd-d27450715d23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" podUID="169c00be-1d02-4314-83bd-d27450715d23"
Oct 30 22:45:08.803080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:08.803054    4581 scope.go:122] "RemoveContainer" containerID="8cb3f87c9d971f02d6822d4daeb4e90eb436a426d48116b6a1052e1f1115adc6"
Oct 30 22:45:08.803523 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:08.803499    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw_odh-ai-gateway-infra(71fc3c7a-b8bd-4e23-a857-09ffb342b6e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3"
Oct 30 22:45:10.821728 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:10.821684    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:45:11.694254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:11.694203    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"]
Oct 30 22:45:11.694465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:11.694442    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy" containerID="cri-o://a59bc00088cf63ff68d40159d80d9848ce12de85864c49fc1175bc0d25c5e667" gracePeriod=30
Oct 30 22:45:15.789152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:15.789116    4581 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:45:15.790271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:15.789181    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:45:16.100073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:16.099989    4581 scope.go:122] "RemoveContainer" containerID="9b4119d0378670ae87f4ed75b7f96ca0f5a70ddade8d9273579a3602f5bbc9f9"
Oct 30 22:45:16.100366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:16.100345    4581 scope.go:122] "RemoveContainer" containerID="ef37f336c205e7f308802440f607b593d849c2063fae498d7cc88c3ee1cf1599"
Oct 30 22:45:16.916540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:16.916462    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" event={"ID":"decc6e06-9261-4213-a87b-26fb98bcbe50","Type":"ContainerStarted","Data":"4bb520d158ea87d37734474b52ab86966d36c441fb390650d28e77b9e4859d3f"}
Oct 30 22:45:16.920286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:16.920251    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" event={"ID":"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f","Type":"ContainerDied","Data":"a59bc00088cf63ff68d40159d80d9848ce12de85864c49fc1175bc0d25c5e667"}
Oct 30 22:45:16.920451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:16.920207    4581 generic.go:356] "Generic (PLEG): container finished" podID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerID="a59bc00088cf63ff68d40159d80d9848ce12de85864c49fc1175bc0d25c5e667" exitCode=0
Oct 30 22:45:16.923250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:16.923177    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" event={"ID":"abeab3fd-da86-4674-943d-0560abd2e752","Type":"ContainerStarted","Data":"f64f449bf65fa2971bc16f01938c011f9332e24acf20dd3bbff2788e7b099976"}
Oct 30 22:45:17.007327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.007302    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:45:17.090593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.090544    4581 scope.go:122] "RemoveContainer" containerID="dc70f0c1e62acf0910e9b62db72f5e1a599446c2b4116fb47f88bb1c8aefcd2c"
Oct 30 22:45:17.109754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.109609    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-envoy\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.109908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.109815    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-podinfo\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.109986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.109948    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istiod-ca-cert\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.110053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.109998    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-socket\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.110053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110039    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-token\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-token\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.110178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110152    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-certs\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.110260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110189    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-kube-api-access-kll56\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-kube-api-access-kll56\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.110318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110255    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-data\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.110318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110306    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-credential-socket\") pod \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\" (UID: \"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f\") "
Oct 30 22:45:17.110506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110378    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-socket" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:17.110506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110405    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istiod-ca-cert" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:17.111132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110633    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-certs" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:17.111132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110849    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-credential-socket" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:17.111132 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.110977    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-data" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:17.111577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.111165    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.111577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.111188    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.111577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.111230    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.111577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.111245    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.111577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.111258    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.113113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.113085    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-podinfo" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:45:17.113565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.113532    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-envoy" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:17.114944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.114918    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-kube-api-access-kll56" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "kube-api-access-kll56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:17.115071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.114959    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-token" pod "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" (UID: "ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:17.213523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.213458    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.213523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.213492    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kll56\" (UniqueName: \"kubernetes.io/projected/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-kube-api-access-kll56\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.213523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.213508    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.213523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.213523    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:17.934495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.934464    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"
Oct 30 22:45:17.934495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.934483    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2" event={"ID":"ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f","Type":"ContainerDied","Data":"c61fee44e288ce72c45a2e234a72372e4f6f16230bd427abb8d695adb1118afa"}
Oct 30 22:45:17.935045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.934544    4581 scope.go:122] "RemoveContainer" containerID="a59bc00088cf63ff68d40159d80d9848ce12de85864c49fc1175bc0d25c5e667"
Oct 30 22:45:17.937170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.937146    4581 generic.go:356] "Generic (PLEG): container finished" podID="abeab3fd-da86-4674-943d-0560abd2e752" containerID="f64f449bf65fa2971bc16f01938c011f9332e24acf20dd3bbff2788e7b099976" exitCode=60
Oct 30 22:45:17.937340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.937272    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" event={"ID":"abeab3fd-da86-4674-943d-0560abd2e752","Type":"ContainerDied","Data":"f64f449bf65fa2971bc16f01938c011f9332e24acf20dd3bbff2788e7b099976"}
Oct 30 22:45:17.937713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.937693    4581 scope.go:122] "RemoveContainer" containerID="f64f449bf65fa2971bc16f01938c011f9332e24acf20dd3bbff2788e7b099976"
Oct 30 22:45:17.938041 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:17.938019    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz_odh-ai-gateway-infra(abeab3fd-da86-4674-943d-0560abd2e752)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" podUID="abeab3fd-da86-4674-943d-0560abd2e752"
Oct 30 22:45:17.939597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.939574    4581 generic.go:356] "Generic (PLEG): container finished" podID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerID="4bb520d158ea87d37734474b52ab86966d36c441fb390650d28e77b9e4859d3f" exitCode=60
Oct 30 22:45:17.939704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.939635    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" event={"ID":"decc6e06-9261-4213-a87b-26fb98bcbe50","Type":"ContainerDied","Data":"4bb520d158ea87d37734474b52ab86966d36c441fb390650d28e77b9e4859d3f"}
Oct 30 22:45:17.940027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.940008    4581 scope.go:122] "RemoveContainer" containerID="4bb520d158ea87d37734474b52ab86966d36c441fb390650d28e77b9e4859d3f"
Oct 30 22:45:17.940368 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:17.940348    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv_odh-ai-gateway-infra(decc6e06-9261-4213-a87b-26fb98bcbe50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50"
Oct 30 22:45:17.941781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.941761    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" event={"ID":"34eb7cbf-d1bd-4141-b0a1-6d7833377c53","Type":"ContainerStarted","Data":"62008142f196dcb143f6fb479867510224c3e8e8715f67968521be72c94e0a2e"}
Oct 30 22:45:17.948439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.948423    4581 scope.go:122] "RemoveContainer" containerID="ef37f336c205e7f308802440f607b593d849c2063fae498d7cc88c3ee1cf1599"
Oct 30 22:45:17.970448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.970412    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"]
Oct 30 22:45:17.973012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.972991    4581 scope.go:122] "RemoveContainer" containerID="9b4119d0378670ae87f4ed75b7f96ca0f5a70ddade8d9273579a3602f5bbc9f9"
Oct 30 22:45:17.995534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:17.995506    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"]
Oct 30 22:45:18.002064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.002037    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-qchs2"]
Oct 30 22:45:18.013688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.013661    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"]
Oct 30 22:45:18.090045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.089433    4581 scope.go:122] "RemoveContainer" containerID="8fa4ff1bbc4e229936105172eb0193e7ea8abfd9d09c0d5e38bf260be2619ed7"
Oct 30 22:45:18.090045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.089535    4581 scope.go:122] "RemoveContainer" containerID="6973e1f5217f60a518eeaa63d5de072d9c76776dd4f270229018ca759a79b4d2"
Oct 30 22:45:18.090045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.089805    4581 scope.go:122] "RemoveContainer" containerID="467bd75710cac9009c50af7c1f64cc2e214732f7e15fe78e6f63d34d6241bdca"
Oct 30 22:45:18.090045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.089904    4581 scope.go:122] "RemoveContainer" containerID="ba428f65f03f84d6fafa15beac1b29f5dfd2a088250c1532b561f7a344944250"
Oct 30 22:45:18.090615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.090565    4581 scope.go:122] "RemoveContainer" containerID="ceb5bd30e0074f63c4459f090513b9912df7f1ddcb0599944232407a87d09378"
Oct 30 22:45:18.094110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.091023    4581 scope.go:122] "RemoveContainer" containerID="bade1104857a54c105235c6a4bf52fa2a1006b78dc1879728c0107a56b37b7c3"
Oct 30 22:45:18.106015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.105983    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" path="/var/lib/kubelet/pods/ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f/volumes"
Oct 30 22:45:18.956195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.956098    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" event={"ID":"23d0420e-107e-40c7-8d92-85c50abb696d","Type":"ContainerStarted","Data":"f579914fe733d9adf383bc3a8da9cde37883d86ea50a56707780ed5eec80c3b0"}
Oct 30 22:45:18.961809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.961779    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" event={"ID":"2296d534-4a85-4b40-a54e-c8b2a67e2aa2","Type":"ContainerStarted","Data":"995e172c89a2346af0e8ef9e137025e3ee712ffed98f0a263a28220508819103"}
Oct 30 22:45:18.964223 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.964191    4581 generic.go:356] "Generic (PLEG): container finished" podID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerID="62008142f196dcb143f6fb479867510224c3e8e8715f67968521be72c94e0a2e" exitCode=60
Oct 30 22:45:18.964360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.964249    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" event={"ID":"34eb7cbf-d1bd-4141-b0a1-6d7833377c53","Type":"ContainerDied","Data":"62008142f196dcb143f6fb479867510224c3e8e8715f67968521be72c94e0a2e"}
Oct 30 22:45:18.964441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.964400    4581 scope.go:122] "RemoveContainer" containerID="dc70f0c1e62acf0910e9b62db72f5e1a599446c2b4116fb47f88bb1c8aefcd2c"
Oct 30 22:45:18.964874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.964854    4581 scope.go:122] "RemoveContainer" containerID="62008142f196dcb143f6fb479867510224c3e8e8715f67968521be72c94e0a2e"
Oct 30 22:45:18.965300 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:18.965276    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr_odh-ai-gateway-infra(34eb7cbf-d1bd-4141-b0a1-6d7833377c53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53"
Oct 30 22:45:18.967412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.967389    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" event={"ID":"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3","Type":"ContainerStarted","Data":"821f4a47384b7c1ef8c5a90ea2d444e38fcc16e91427ba4224c70f5b2d1be658"}
Oct 30 22:45:18.970153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.970127    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" event={"ID":"e93290c9-59d9-48e6-990a-b18d680913ed","Type":"ContainerStarted","Data":"52cb90df9879cb084eb5772c7a6eac582b05e965e08be63e1c984fb602247c27"}
Oct 30 22:45:18.974312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.974283    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" event={"ID":"abfeb292-6e71-4543-9215-a445f8226c13","Type":"ContainerStarted","Data":"751ba7a5b07aa3381e06387d18c03b91fdf151a9e0f0ca802d3eee89079ae978"}
Oct 30 22:45:18.976700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:18.976676    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" event={"ID":"31f18846-74ba-48c9-a797-0f33d1820e16","Type":"ContainerStarted","Data":"546855bc56e82aa2bb719c57bf25920a92839bd2b46a09a151f06f7cf6ef377f"}
Oct 30 22:45:19.033944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.033882    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" podStartSLOduration=19.033864658 podStartE2EDuration="19.033864658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:19.027008262 +0000 UTC m=+2303.065947145" watchObservedRunningTime="2026-10-30 22:45:19.033864658 +0000 UTC m=+2303.072803545"
Oct 30 22:45:19.089279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.086664    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"]
Oct 30 22:45:19.089668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.089641    4581 scope.go:122] "RemoveContainer" containerID="8cb3f87c9d971f02d6822d4daeb4e90eb436a426d48116b6a1052e1f1115adc6"
Oct 30 22:45:19.091945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.091830    4581 scope.go:122] "RemoveContainer" containerID="049bb9121132cf5ab3a110fc7ebcb43fbfa6f2a6c565f672dbc1e099d80a3f2a"
Oct 30 22:45:19.094996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.094146    4581 scope.go:122] "RemoveContainer" containerID="23e159785f52ff901070c28f0ea97489d7074fe92b843e093b1e550bc35d80de"
Oct 30 22:45:19.094996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.094631    4581 scope.go:122] "RemoveContainer" containerID="a88a46e8c77268a102b17e437b0ce5ba3b88d44aeb67c3a4b529c359a7a01913"
Oct 30 22:45:19.244537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.244453    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:19.397930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.373603    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decc6e06-9261-4213-a87b-26fb98bcbe50-kube-api-access-d2dnq\" (UniqueName: \"kubernetes.io/projected/decc6e06-9261-4213-a87b-26fb98bcbe50-kube-api-access-d2dnq\") pod \"decc6e06-9261-4213-a87b-26fb98bcbe50\" (UID: \"decc6e06-9261-4213-a87b-26fb98bcbe50\") "
Oct 30 22:45:19.397930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.374009    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decc6e06-9261-4213-a87b-26fb98bcbe50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decc6e06-9261-4213-a87b-26fb98bcbe50-maas-api-service-ca\") pod \"decc6e06-9261-4213-a87b-26fb98bcbe50\" (UID: \"decc6e06-9261-4213-a87b-26fb98bcbe50\") "
Oct 30 22:45:19.397930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.378031    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decc6e06-9261-4213-a87b-26fb98bcbe50-maas-api-service-ca" pod "decc6e06-9261-4213-a87b-26fb98bcbe50" (UID: "decc6e06-9261-4213-a87b-26fb98bcbe50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.397930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.383763    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decc6e06-9261-4213-a87b-26fb98bcbe50-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:19.462475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.462424    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decc6e06-9261-4213-a87b-26fb98bcbe50-kube-api-access-d2dnq" pod "decc6e06-9261-4213-a87b-26fb98bcbe50" (UID: "decc6e06-9261-4213-a87b-26fb98bcbe50"). InnerVolumeSpecName "kube-api-access-d2dnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.485775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.485734    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2dnq\" (UniqueName: \"kubernetes.io/projected/decc6e06-9261-4213-a87b-26fb98bcbe50-kube-api-access-d2dnq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:19.720406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.720380    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:19.791507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.790313    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abeab3fd-da86-4674-943d-0560abd2e752-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abeab3fd-da86-4674-943d-0560abd2e752-maas-api-service-ca\") pod \"abeab3fd-da86-4674-943d-0560abd2e752\" (UID: \"abeab3fd-da86-4674-943d-0560abd2e752\") "
Oct 30 22:45:19.791507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.790495    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abeab3fd-da86-4674-943d-0560abd2e752-kube-api-access-fls6n\" (UniqueName: \"kubernetes.io/projected/abeab3fd-da86-4674-943d-0560abd2e752-kube-api-access-fls6n\") pod \"abeab3fd-da86-4674-943d-0560abd2e752\" (UID: \"abeab3fd-da86-4674-943d-0560abd2e752\") "
Oct 30 22:45:19.795479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.795443    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abeab3fd-da86-4674-943d-0560abd2e752-maas-api-service-ca" pod "abeab3fd-da86-4674-943d-0560abd2e752" (UID: "abeab3fd-da86-4674-943d-0560abd2e752"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.855479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.855433    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abeab3fd-da86-4674-943d-0560abd2e752-kube-api-access-fls6n" pod "abeab3fd-da86-4674-943d-0560abd2e752" (UID: "abeab3fd-da86-4674-943d-0560abd2e752"). InnerVolumeSpecName "kube-api-access-fls6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.893319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.893278    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fls6n\" (UniqueName: \"kubernetes.io/projected/abeab3fd-da86-4674-943d-0560abd2e752-kube-api-access-fls6n\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:19.893498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:19.893329    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abeab3fd-da86-4674-943d-0560abd2e752-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.010311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.010278    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"]
Oct 30 22:45:20.011060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.011034    4581 generic.go:356] "Generic (PLEG): container finished" podID="23d0420e-107e-40c7-8d92-85c50abb696d" containerID="f579914fe733d9adf383bc3a8da9cde37883d86ea50a56707780ed5eec80c3b0" exitCode=60
Oct 30 22:45:20.011267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.011116    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" event={"ID":"23d0420e-107e-40c7-8d92-85c50abb696d","Type":"ContainerDied","Data":"f579914fe733d9adf383bc3a8da9cde37883d86ea50a56707780ed5eec80c3b0"}
Oct 30 22:45:20.011267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.011186    4581 scope.go:122] "RemoveContainer" containerID="8fa4ff1bbc4e229936105172eb0193e7ea8abfd9d09c0d5e38bf260be2619ed7"
Oct 30 22:45:20.015029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.014989    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"
Oct 30 22:45:20.023954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.023852    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv" event={"ID":"decc6e06-9261-4213-a87b-26fb98bcbe50","Type":"ContainerDied","Data":"421e5b7b18bda526bdd45bf3e5700fe31793fb1c02e6b19a3a9bb2bed28843af"}
Oct 30 22:45:20.027158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.027126    4581 generic.go:356] "Generic (PLEG): container finished" podID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerID="995e172c89a2346af0e8ef9e137025e3ee712ffed98f0a263a28220508819103" exitCode=60
Oct 30 22:45:20.027310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.027295    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" event={"ID":"2296d534-4a85-4b40-a54e-c8b2a67e2aa2","Type":"ContainerDied","Data":"995e172c89a2346af0e8ef9e137025e3ee712ffed98f0a263a28220508819103"}
Oct 30 22:45:20.027983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.027778    4581 scope.go:122] "RemoveContainer" containerID="995e172c89a2346af0e8ef9e137025e3ee712ffed98f0a263a28220508819103"
Oct 30 22:45:20.028410 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:20.028244    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85_odh-ai-gateway-infra(2296d534-4a85-4b40-a54e-c8b2a67e2aa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2"
Oct 30 22:45:20.030595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.030573    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" event={"ID":"fd40171c-a1f5-484c-a078-ce72ef26cb90","Type":"ContainerStarted","Data":"d3894049e6014e14cd430474176200f991408d529723df14a15da35a7485af30"}
Oct 30 22:45:20.036230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.036196    4581 scope.go:122] "RemoveContainer" containerID="4bb520d158ea87d37734474b52ab86966d36c441fb390650d28e77b9e4859d3f"
Oct 30 22:45:20.036713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.036690    4581 generic.go:356] "Generic (PLEG): container finished" podID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerID="821f4a47384b7c1ef8c5a90ea2d444e38fcc16e91427ba4224c70f5b2d1be658" exitCode=60
Oct 30 22:45:20.036905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.036734    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" event={"ID":"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3","Type":"ContainerDied","Data":"821f4a47384b7c1ef8c5a90ea2d444e38fcc16e91427ba4224c70f5b2d1be658"}
Oct 30 22:45:20.038656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.038360    4581 scope.go:122] "RemoveContainer" containerID="821f4a47384b7c1ef8c5a90ea2d444e38fcc16e91427ba4224c70f5b2d1be658"
Oct 30 22:45:20.038873 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:20.038832    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5_odh-ai-gateway-infra(8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3"
Oct 30 22:45:20.040761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.040736    4581 generic.go:356] "Generic (PLEG): container finished" podID="e93290c9-59d9-48e6-990a-b18d680913ed" containerID="52cb90df9879cb084eb5772c7a6eac582b05e965e08be63e1c984fb602247c27" exitCode=60
Oct 30 22:45:20.040886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.040802    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" event={"ID":"e93290c9-59d9-48e6-990a-b18d680913ed","Type":"ContainerDied","Data":"52cb90df9879cb084eb5772c7a6eac582b05e965e08be63e1c984fb602247c27"}
Oct 30 22:45:20.041323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.041287    4581 scope.go:122] "RemoveContainer" containerID="52cb90df9879cb084eb5772c7a6eac582b05e965e08be63e1c984fb602247c27"
Oct 30 22:45:20.041776 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:20.041712    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq_odh-ai-gateway-infra(e93290c9-59d9-48e6-990a-b18d680913ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" podUID="e93290c9-59d9-48e6-990a-b18d680913ed"
Oct 30 22:45:20.043769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.043738    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" event={"ID":"169c00be-1d02-4314-83bd-d27450715d23","Type":"ContainerStarted","Data":"1f824563df1e677946143c7301d733ba8d010763653540e1afbc84ca3adcc6d0"}
Oct 30 22:45:20.046479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.046398    4581 generic.go:356] "Generic (PLEG): container finished" podID="abfeb292-6e71-4543-9215-a445f8226c13" containerID="751ba7a5b07aa3381e06387d18c03b91fdf151a9e0f0ca802d3eee89079ae978" exitCode=60
Oct 30 22:45:20.046581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.046565    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" event={"ID":"abfeb292-6e71-4543-9215-a445f8226c13","Type":"ContainerDied","Data":"751ba7a5b07aa3381e06387d18c03b91fdf151a9e0f0ca802d3eee89079ae978"}
Oct 30 22:45:20.046965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.046945    4581 scope.go:122] "RemoveContainer" containerID="751ba7a5b07aa3381e06387d18c03b91fdf151a9e0f0ca802d3eee89079ae978"
Oct 30 22:45:20.047496 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:20.047466    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b_odh-ai-gateway-infra(abfeb292-6e71-4543-9215-a445f8226c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" podUID="abfeb292-6e71-4543-9215-a445f8226c13"
Oct 30 22:45:20.047788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.047763    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"]
Oct 30 22:45:20.053235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.053098    4581 generic.go:356] "Generic (PLEG): container finished" podID="31f18846-74ba-48c9-a797-0f33d1820e16" containerID="546855bc56e82aa2bb719c57bf25920a92839bd2b46a09a151f06f7cf6ef377f" exitCode=60
Oct 30 22:45:20.053627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.053560    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" event={"ID":"31f18846-74ba-48c9-a797-0f33d1820e16","Type":"ContainerDied","Data":"546855bc56e82aa2bb719c57bf25920a92839bd2b46a09a151f06f7cf6ef377f"}
Oct 30 22:45:20.056893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.056394    4581 scope.go:122] "RemoveContainer" containerID="546855bc56e82aa2bb719c57bf25920a92839bd2b46a09a151f06f7cf6ef377f"
Oct 30 22:45:20.056893 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:20.056818    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8_odh-ai-gateway-infra(31f18846-74ba-48c9-a797-0f33d1820e16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" podUID="31f18846-74ba-48c9-a797-0f33d1820e16"
Oct 30 22:45:20.069697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.068777    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" event={"ID":"a5f7488c-2538-4dc6-818b-306e295933cc","Type":"ContainerStarted","Data":"4781ebd4c4f3da40fc799aa078451f96da14c1cd9333cf70cf9528418e67b870"}
Oct 30 22:45:20.073089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.072927    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz" event={"ID":"abeab3fd-da86-4674-943d-0560abd2e752","Type":"ContainerDied","Data":"c4293e3fa3220d8947f8b4df62e76cbd812ccbf71ca8f924bfe1fd26d985606b"}
Oct 30 22:45:20.073089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.073033    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"
Oct 30 22:45:20.073369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.073350    4581 scope.go:122] "RemoveContainer" containerID="ceb5bd30e0074f63c4459f090513b9912df7f1ddcb0599944232407a87d09378"
Oct 30 22:45:20.092637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.090667    4581 scope.go:122] "RemoveContainer" containerID="5c11f93624efc5a8546ba61d8bfbe2277e1e182070beb541d63af2ff6a6c23f4"
Oct 30 22:45:20.092637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.090883    4581 scope.go:122] "RemoveContainer" containerID="e0494094f2ce291b277e2cbf7bcf545dac115b0ab5bee86a938f5e0214d4b543"
Oct 30 22:45:20.092637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.092590    4581 scope.go:122] "RemoveContainer" containerID="52ee624532f686244f779cbd0262edcf521de8d6f77954778a8321fd146e26d7"
Oct 30 22:45:20.093663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.093277    4581 scope.go:122] "RemoveContainer" containerID="fb8d3ae0654b0cbbbcc9a2b2fa2dbf54a754609d87d2965c8b9db797fcbc37e7"
Oct 30 22:45:20.150475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.150376    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"]
Oct 30 22:45:20.166507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.166485    4581 scope.go:122] "RemoveContainer" containerID="ba428f65f03f84d6fafa15beac1b29f5dfd2a088250c1532b561f7a344944250"
Oct 30 22:45:20.314400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.314278    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"]
Oct 30 22:45:20.337733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.337693    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"]
Oct 30 22:45:20.346913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.346889    4581 scope.go:122] "RemoveContainer" containerID="bade1104857a54c105235c6a4bf52fa2a1006b78dc1879728c0107a56b37b7c3"
Oct 30 22:45:20.362008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.361533    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:20.369527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.365451    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"]
Oct 30 22:45:20.401503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.401470    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"]
Oct 30 22:45:20.407075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.407036    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890005-lm7sv"]
Oct 30 22:45:20.423872 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.423851    4581 scope.go:122] "RemoveContainer" containerID="6973e1f5217f60a518eeaa63d5de072d9c76776dd4f270229018ca759a79b4d2"
Oct 30 22:45:20.425872 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.425852    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:20.439897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.439877    4581 scope.go:122] "RemoveContainer" containerID="467bd75710cac9009c50af7c1f64cc2e214732f7e15fe78e6f63d34d6241bdca"
Oct 30 22:45:20.459124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.459083    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d0420e-107e-40c7-8d92-85c50abb696d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0420e-107e-40c7-8d92-85c50abb696d-maas-api-service-ca\") pod \"23d0420e-107e-40c7-8d92-85c50abb696d\" (UID: \"23d0420e-107e-40c7-8d92-85c50abb696d\") "
Oct 30 22:45:20.459361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.459318    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d0420e-107e-40c7-8d92-85c50abb696d-kube-api-access-vkvcz\" (UniqueName: \"kubernetes.io/projected/23d0420e-107e-40c7-8d92-85c50abb696d-kube-api-access-vkvcz\") pod \"23d0420e-107e-40c7-8d92-85c50abb696d\" (UID: \"23d0420e-107e-40c7-8d92-85c50abb696d\") "
Oct 30 22:45:20.459614 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.459593    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-maas-api-service-ca\") pod \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\" (UID: \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\") "
Oct 30 22:45:20.459806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.459767    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-kube-api-access-dbls7\" (UniqueName: \"kubernetes.io/projected/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-kube-api-access-dbls7\") pod \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\" (UID: \"34eb7cbf-d1bd-4141-b0a1-6d7833377c53\") "
Oct 30 22:45:20.460754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.460715    4581 scope.go:122] "RemoveContainer" containerID="f64f449bf65fa2971bc16f01938c011f9332e24acf20dd3bbff2788e7b099976"
Oct 30 22:45:20.462138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.461538    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-maas-api-service-ca" pod "34eb7cbf-d1bd-4141-b0a1-6d7833377c53" (UID: "34eb7cbf-d1bd-4141-b0a1-6d7833377c53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.462138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.462099    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d0420e-107e-40c7-8d92-85c50abb696d-maas-api-service-ca" pod "23d0420e-107e-40c7-8d92-85c50abb696d" (UID: "23d0420e-107e-40c7-8d92-85c50abb696d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.512453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.512351    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"]
Oct 30 22:45:20.517962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.517934    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890005-bh8kz"]
Oct 30 22:45:20.561596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.559702    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-kube-api-access-dbls7" pod "34eb7cbf-d1bd-4141-b0a1-6d7833377c53" (UID: "34eb7cbf-d1bd-4141-b0a1-6d7833377c53"). InnerVolumeSpecName "kube-api-access-dbls7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.561596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.560449    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d0420e-107e-40c7-8d92-85c50abb696d-kube-api-access-vkvcz" pod "23d0420e-107e-40c7-8d92-85c50abb696d" (UID: "23d0420e-107e-40c7-8d92-85c50abb696d"). InnerVolumeSpecName "kube-api-access-vkvcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.562362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.562276    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbls7\" (UniqueName: \"kubernetes.io/projected/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-kube-api-access-dbls7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.562362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.562308    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d0420e-107e-40c7-8d92-85c50abb696d-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.562362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.562324    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkvcz\" (UniqueName: \"kubernetes.io/projected/23d0420e-107e-40c7-8d92-85c50abb696d-kube-api-access-vkvcz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:20.562362 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:20.562338    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34eb7cbf-d1bd-4141-b0a1-6d7833377c53-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.087310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.087272    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" event={"ID":"071439a8-bb97-4ab1-b4af-e0c88e7a585e","Type":"ContainerStarted","Data":"ca4373f3b4bc6ff85174a8ae1214711d33a3396499aa41c4ee84301d1d8826d7"}
Oct 30 22:45:21.100391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.092414    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" event={"ID":"dd3f411a-89df-4177-bb36-b9bdcca5e8c2","Type":"ContainerStarted","Data":"6cc8a555dda469569062d400d3c06f05f05e34d323aecc227c7aaf61331853d3"}
Oct 30 22:45:21.100391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.095916    4581 generic.go:356] "Generic (PLEG): container finished" podID="a5f7488c-2538-4dc6-818b-306e295933cc" containerID="4781ebd4c4f3da40fc799aa078451f96da14c1cd9333cf70cf9528418e67b870" exitCode=60
Oct 30 22:45:21.100391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.096079    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" event={"ID":"a5f7488c-2538-4dc6-818b-306e295933cc","Type":"ContainerDied","Data":"4781ebd4c4f3da40fc799aa078451f96da14c1cd9333cf70cf9528418e67b870"}
Oct 30 22:45:21.100391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.096190    4581 scope.go:122] "RemoveContainer" containerID="a88a46e8c77268a102b17e437b0ce5ba3b88d44aeb67c3a4b529c359a7a01913"
Oct 30 22:45:21.100391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.096845    4581 scope.go:122] "RemoveContainer" containerID="4781ebd4c4f3da40fc799aa078451f96da14c1cd9333cf70cf9528418e67b870"
Oct 30 22:45:21.100391 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:21.099117    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw_odh-ai-gateway-infra(a5f7488c-2538-4dc6-818b-306e295933cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" podUID="a5f7488c-2538-4dc6-818b-306e295933cc"
Oct 30 22:45:21.107095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.106131    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" event={"ID":"fb334b3a-3f29-4505-ae06-d4ce0917bebc","Type":"ContainerStarted","Data":"8cc60b757662a3ac0968cc909b9013b2edddeae9d0845c0bdae5ecbfc83b6b04"}
Oct 30 22:45:21.115973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.114874    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" event={"ID":"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb","Type":"ContainerStarted","Data":"df793d5f88d43714879409824f5d6f0958ced0ddc18e5e772f2a9f93c64dfcc7"}
Oct 30 22:45:21.151590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.118651    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw" event={"ID":"23d0420e-107e-40c7-8d92-85c50abb696d","Type":"ContainerDied","Data":"e60d4a05a243a1a4d88bc2161464e159d4c013dd2723f9976b16c4e9954e8dda"}
Oct 30 22:45:21.151590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.118801    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"
Oct 30 22:45:21.211780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.211203    4581 scope.go:122] "RemoveContainer" containerID="f579914fe733d9adf383bc3a8da9cde37883d86ea50a56707780ed5eec80c3b0"
Oct 30 22:45:21.215281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.215142    4581 generic.go:356] "Generic (PLEG): container finished" podID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerID="d3894049e6014e14cd430474176200f991408d529723df14a15da35a7485af30" exitCode=60
Oct 30 22:45:21.217606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.215499    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" event={"ID":"fd40171c-a1f5-484c-a078-ce72ef26cb90","Type":"ContainerDied","Data":"d3894049e6014e14cd430474176200f991408d529723df14a15da35a7485af30"}
Oct 30 22:45:21.217606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.216056    4581 scope.go:122] "RemoveContainer" containerID="d3894049e6014e14cd430474176200f991408d529723df14a15da35a7485af30"
Oct 30 22:45:21.217606 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:21.216605    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d_odh-ai-gateway-infra(fd40171c-a1f5-484c-a078-ce72ef26cb90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90"
Oct 30 22:45:21.220814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.219891    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"
Oct 30 22:45:21.221305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.221280    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr" event={"ID":"34eb7cbf-d1bd-4141-b0a1-6d7833377c53","Type":"ContainerDied","Data":"c68772e8d6b2642c62ffff76abc68466c1bbe4f4301d9db5ca6979b1c81ad14a"}
Oct 30 22:45:21.222685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.222664    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"]
Oct 30 22:45:21.236828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.236802    4581 scope.go:122] "RemoveContainer" containerID="049bb9121132cf5ab3a110fc7ebcb43fbfa6f2a6c565f672dbc1e099d80a3f2a"
Oct 30 22:45:21.238047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.238017    4581 generic.go:356] "Generic (PLEG): container finished" podID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerID="582f0c196b8cab5494e4fdbbdac541f3b7778fee9ab6b7a96e4a46ac77e07f19" exitCode=60
Oct 30 22:45:21.238555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.238506    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" event={"ID":"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3","Type":"ContainerDied","Data":"582f0c196b8cab5494e4fdbbdac541f3b7778fee9ab6b7a96e4a46ac77e07f19"}
Oct 30 22:45:21.239015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.238987    4581 scope.go:122] "RemoveContainer" containerID="582f0c196b8cab5494e4fdbbdac541f3b7778fee9ab6b7a96e4a46ac77e07f19"
Oct 30 22:45:21.241334 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:21.240584    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw_odh-ai-gateway-infra(71fc3c7a-b8bd-4e23-a857-09ffb342b6e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3"
Oct 30 22:45:21.244389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.244365    4581 generic.go:356] "Generic (PLEG): container finished" podID="169c00be-1d02-4314-83bd-d27450715d23" containerID="1f824563df1e677946143c7301d733ba8d010763653540e1afbc84ca3adcc6d0" exitCode=60
Oct 30 22:45:21.244688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.244530    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" event={"ID":"169c00be-1d02-4314-83bd-d27450715d23","Type":"ContainerDied","Data":"1f824563df1e677946143c7301d733ba8d010763653540e1afbc84ca3adcc6d0"}
Oct 30 22:45:21.245659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.245643    4581 scope.go:122] "RemoveContainer" containerID="1f824563df1e677946143c7301d733ba8d010763653540e1afbc84ca3adcc6d0"
Oct 30 22:45:21.246651 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:21.246580    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm_odh-ai-gateway-infra(169c00be-1d02-4314-83bd-d27450715d23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" podUID="169c00be-1d02-4314-83bd-d27450715d23"
Oct 30 22:45:21.341463 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.341143    4581 scope.go:122] "RemoveContainer" containerID="62008142f196dcb143f6fb479867510224c3e8e8715f67968521be72c94e0a2e"
Oct 30 22:45:21.380008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.379978    4581 scope.go:122] "RemoveContainer" containerID="8cb3f87c9d971f02d6822d4daeb4e90eb436a426d48116b6a1052e1f1115adc6"
Oct 30 22:45:21.431904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.431872    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"]
Oct 30 22:45:21.461359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.461327    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"]
Oct 30 22:45:21.470609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.470386    4581 scope.go:122] "RemoveContainer" containerID="23e159785f52ff901070c28f0ea97489d7074fe92b843e093b1e550bc35d80de"
Oct 30 22:45:21.470927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.470519    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890005-drpwr"]
Oct 30 22:45:21.474391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.474362    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:21.492483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.492393    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"]
Oct 30 22:45:21.513255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.511759    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-999470-29890005-snxmw"]
Oct 30 22:45:21.513255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.513062    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f18846-74ba-48c9-a797-0f33d1820e16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f18846-74ba-48c9-a797-0f33d1820e16-maas-api-service-ca\") pod \"31f18846-74ba-48c9-a797-0f33d1820e16\" (UID: \"31f18846-74ba-48c9-a797-0f33d1820e16\") "
Oct 30 22:45:21.513255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.513195    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f18846-74ba-48c9-a797-0f33d1820e16-kube-api-access-7fkxr\" (UniqueName: \"kubernetes.io/projected/31f18846-74ba-48c9-a797-0f33d1820e16-kube-api-access-7fkxr\") pod \"31f18846-74ba-48c9-a797-0f33d1820e16\" (UID: \"31f18846-74ba-48c9-a797-0f33d1820e16\") "
Oct 30 22:45:21.520343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.518338    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f18846-74ba-48c9-a797-0f33d1820e16-maas-api-service-ca" pod "31f18846-74ba-48c9-a797-0f33d1820e16" (UID: "31f18846-74ba-48c9-a797-0f33d1820e16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:21.569454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.569417    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f18846-74ba-48c9-a797-0f33d1820e16-kube-api-access-7fkxr" pod "31f18846-74ba-48c9-a797-0f33d1820e16" (UID: "31f18846-74ba-48c9-a797-0f33d1820e16"). InnerVolumeSpecName "kube-api-access-7fkxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.612465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.612009    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"]
Oct 30 22:45:21.618784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.618752    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f18846-74ba-48c9-a797-0f33d1820e16-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.618784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.618785    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fkxr\" (UniqueName: \"kubernetes.io/projected/31f18846-74ba-48c9-a797-0f33d1820e16-kube-api-access-7fkxr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.671170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.671138    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:21.675395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.675370    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:21.719518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.719484    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-maas-api-service-ca\") pod \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\" (UID: \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\") "
Oct 30 22:45:21.719728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.719606    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-kube-api-access-787sf\" (UniqueName: \"kubernetes.io/projected/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-kube-api-access-787sf\") pod \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\" (UID: \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\") "
Oct 30 22:45:21.719728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.719644    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-maas-api-service-ca\") pod \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\" (UID: \"2296d534-4a85-4b40-a54e-c8b2a67e2aa2\") "
Oct 30 22:45:21.720239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.719860    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-maas-api-service-ca" pod "8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" (UID: "8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:21.720239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.719915    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-kube-api-access-kldr7\" (UniqueName: \"kubernetes.io/projected/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-kube-api-access-kldr7\") pod \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\" (UID: \"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3\") "
Oct 30 22:45:21.720657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.720636    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.721044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.721016    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-maas-api-service-ca" pod "2296d534-4a85-4b40-a54e-c8b2a67e2aa2" (UID: "2296d534-4a85-4b40-a54e-c8b2a67e2aa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:21.751818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.751699    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-kube-api-access-787sf" pod "2296d534-4a85-4b40-a54e-c8b2a67e2aa2" (UID: "2296d534-4a85-4b40-a54e-c8b2a67e2aa2"). InnerVolumeSpecName "kube-api-access-787sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.751818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.751787    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-kube-api-access-kldr7" pod "8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" (UID: "8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3"). InnerVolumeSpecName "kube-api-access-kldr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.822768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.822737    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kldr7\" (UniqueName: \"kubernetes.io/projected/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3-kube-api-access-kldr7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.822768 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.822767    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-787sf\" (UniqueName: \"kubernetes.io/projected/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-kube-api-access-787sf\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.822978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.822776    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2296d534-4a85-4b40-a54e-c8b2a67e2aa2-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.873130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.873053    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:21.879228 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.879173    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.923431    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93290c9-59d9-48e6-990a-b18d680913ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93290c9-59d9-48e6-990a-b18d680913ed-maas-api-service-ca\") pod \"e93290c9-59d9-48e6-990a-b18d680913ed\" (UID: \"e93290c9-59d9-48e6-990a-b18d680913ed\") "
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.923749    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfeb292-6e71-4543-9215-a445f8226c13-kube-api-access-x7g7b\" (UniqueName: \"kubernetes.io/projected/abfeb292-6e71-4543-9215-a445f8226c13-kube-api-access-x7g7b\") pod \"abfeb292-6e71-4543-9215-a445f8226c13\" (UID: \"abfeb292-6e71-4543-9215-a445f8226c13\") "
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.923874    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93290c9-59d9-48e6-990a-b18d680913ed-maas-api-service-ca" pod "e93290c9-59d9-48e6-990a-b18d680913ed" (UID: "e93290c9-59d9-48e6-990a-b18d680913ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.923896    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abfeb292-6e71-4543-9215-a445f8226c13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfeb292-6e71-4543-9215-a445f8226c13-maas-api-service-ca\") pod \"abfeb292-6e71-4543-9215-a445f8226c13\" (UID: \"abfeb292-6e71-4543-9215-a445f8226c13\") "
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.923970    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93290c9-59d9-48e6-990a-b18d680913ed-kube-api-access-kj75m\" (UniqueName: \"kubernetes.io/projected/e93290c9-59d9-48e6-990a-b18d680913ed-kube-api-access-kj75m\") pod \"e93290c9-59d9-48e6-990a-b18d680913ed\" (UID: \"e93290c9-59d9-48e6-990a-b18d680913ed\") "
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.924353    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfeb292-6e71-4543-9215-a445f8226c13-maas-api-service-ca" pod "abfeb292-6e71-4543-9215-a445f8226c13" (UID: "abfeb292-6e71-4543-9215-a445f8226c13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.924718    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93290c9-59d9-48e6-990a-b18d680913ed-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.924740    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfeb292-6e71-4543-9215-a445f8226c13-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.926970    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93290c9-59d9-48e6-990a-b18d680913ed-kube-api-access-kj75m" pod "e93290c9-59d9-48e6-990a-b18d680913ed" (UID: "e93290c9-59d9-48e6-990a-b18d680913ed"). InnerVolumeSpecName "kube-api-access-kj75m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.927301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:21.927009    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfeb292-6e71-4543-9215-a445f8226c13-kube-api-access-x7g7b" pod "abfeb292-6e71-4543-9215-a445f8226c13" (UID: "abfeb292-6e71-4543-9215-a445f8226c13"). InnerVolumeSpecName "kube-api-access-x7g7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.026739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.026701    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj75m\" (UniqueName: \"kubernetes.io/projected/e93290c9-59d9-48e6-990a-b18d680913ed-kube-api-access-kj75m\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.026739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.026738    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7g7b\" (UniqueName: \"kubernetes.io/projected/abfeb292-6e71-4543-9215-a445f8226c13-kube-api-access-x7g7b\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.106703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.106672    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" path="/var/lib/kubelet/pods/23d0420e-107e-40c7-8d92-85c50abb696d/volumes"
Oct 30 22:45:22.107185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.107170    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" path="/var/lib/kubelet/pods/34eb7cbf-d1bd-4141-b0a1-6d7833377c53/volumes"
Oct 30 22:45:22.107631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.107617    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abeab3fd-da86-4674-943d-0560abd2e752" path="/var/lib/kubelet/pods/abeab3fd-da86-4674-943d-0560abd2e752/volumes"
Oct 30 22:45:22.108328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.108313    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" path="/var/lib/kubelet/pods/decc6e06-9261-4213-a87b-26fb98bcbe50/volumes"
Oct 30 22:45:22.134635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.134605    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"]
Oct 30 22:45:22.262290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.262244    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"
Oct 30 22:45:22.262290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.262282    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85" event={"ID":"2296d534-4a85-4b40-a54e-c8b2a67e2aa2","Type":"ContainerDied","Data":"29749b6c02842061d204f8669e398875b6cf511e804c7d27882a84da63de64ad"}
Oct 30 22:45:22.262561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.262364    4581 scope.go:122] "RemoveContainer" containerID="995e172c89a2346af0e8ef9e137025e3ee712ffed98f0a263a28220508819103"
Oct 30 22:45:22.268135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.268111    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"
Oct 30 22:45:22.268311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.268137    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5" event={"ID":"8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3","Type":"ContainerDied","Data":"22b297a97760762c6b7397b94d0331ef138b5228687f7607f22e23028edee424"}
Oct 30 22:45:22.269934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.269915    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"
Oct 30 22:45:22.270050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.269932    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq" event={"ID":"e93290c9-59d9-48e6-990a-b18d680913ed","Type":"ContainerDied","Data":"a1160ddde82238e1631478f39b64f0116a7f5c779db3a7d6bc28f034eaddadf3"}
Oct 30 22:45:22.280963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.280932    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b" event={"ID":"abfeb292-6e71-4543-9215-a445f8226c13","Type":"ContainerDied","Data":"e552ae831591ad2a8986bddc593ed0e8a0fc0a9aae9ca07f2e3c3b372c9f1586"}
Oct 30 22:45:22.281160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.280995    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"
Oct 30 22:45:22.282359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.282167    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"]
Oct 30 22:45:22.285074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.285052    4581 scope.go:122] "RemoveContainer" containerID="821f4a47384b7c1ef8c5a90ea2d444e38fcc16e91427ba4224c70f5b2d1be658"
Oct 30 22:45:22.286098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.286040    4581 generic.go:356] "Generic (PLEG): container finished" podID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerID="ca4373f3b4bc6ff85174a8ae1214711d33a3396499aa41c4ee84301d1d8826d7" exitCode=60
Oct 30 22:45:22.286296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.286171    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" event={"ID":"071439a8-bb97-4ab1-b4af-e0c88e7a585e","Type":"ContainerDied","Data":"ca4373f3b4bc6ff85174a8ae1214711d33a3396499aa41c4ee84301d1d8826d7"}
Oct 30 22:45:22.297612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.297581    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8" event={"ID":"31f18846-74ba-48c9-a797-0f33d1820e16","Type":"ContainerDied","Data":"d6be1c9d47fcf84771833730596193283c5c542f70f27697105afc2f085ac23c"}
Oct 30 22:45:22.301879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.298132    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"
Oct 30 22:45:22.304588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.304557    4581 generic.go:356] "Generic (PLEG): container finished" podID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerID="6cc8a555dda469569062d400d3c06f05f05e34d323aecc227c7aaf61331853d3" exitCode=60
Oct 30 22:45:22.304702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.304683    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" event={"ID":"dd3f411a-89df-4177-bb36-b9bdcca5e8c2","Type":"ContainerDied","Data":"6cc8a555dda469569062d400d3c06f05f05e34d323aecc227c7aaf61331853d3"}
Oct 30 22:45:22.305179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.305160    4581 scope.go:122] "RemoveContainer" containerID="6cc8a555dda469569062d400d3c06f05f05e34d323aecc227c7aaf61331853d3"
Oct 30 22:45:22.305627 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:22.305603    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v_odh-ai-gateway-infra(dd3f411a-89df-4177-bb36-b9bdcca5e8c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2"
Oct 30 22:45:22.306628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.306206    4581 scope.go:122] "RemoveContainer" containerID="52cb90df9879cb084eb5772c7a6eac582b05e965e08be63e1c984fb602247c27"
Oct 30 22:45:22.318265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.317340    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"]
Oct 30 22:45:22.323265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.322375    4581 generic.go:356] "Generic (PLEG): container finished" podID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerID="8cc60b757662a3ac0968cc909b9013b2edddeae9d0845c0bdae5ecbfc83b6b04" exitCode=60
Oct 30 22:45:22.323265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.322461    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" event={"ID":"fb334b3a-3f29-4505-ae06-d4ce0917bebc","Type":"ContainerDied","Data":"8cc60b757662a3ac0968cc909b9013b2edddeae9d0845c0bdae5ecbfc83b6b04"}
Oct 30 22:45:22.326839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.326815    4581 scope.go:122] "RemoveContainer" containerID="751ba7a5b07aa3381e06387d18c03b91fdf151a9e0f0ca802d3eee89079ae978"
Oct 30 22:45:22.328809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.328785    4581 generic.go:356] "Generic (PLEG): container finished" podID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerID="df793d5f88d43714879409824f5d6f0958ced0ddc18e5e772f2a9f93c64dfcc7" exitCode=60
Oct 30 22:45:22.328913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.328861    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" event={"ID":"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb","Type":"ContainerDied","Data":"df793d5f88d43714879409824f5d6f0958ced0ddc18e5e772f2a9f93c64dfcc7"}
Oct 30 22:45:22.329562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.329539    4581 scope.go:122] "RemoveContainer" containerID="df793d5f88d43714879409824f5d6f0958ced0ddc18e5e772f2a9f93c64dfcc7"
Oct 30 22:45:22.330027 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:22.329997    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc_odh-ai-gateway-infra(973aa012-ec8e-455f-9c5a-4fd55e3d4dbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb"
Oct 30 22:45:22.411654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.411630    4581 scope.go:122] "RemoveContainer" containerID="5c11f93624efc5a8546ba61d8bfbe2277e1e182070beb541d63af2ff6a6c23f4"
Oct 30 22:45:22.423530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.422878    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"]
Oct 30 22:45:22.431273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.431243    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bc48cc-29890005-l6vj5"]
Oct 30 22:45:22.439074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.438955    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"]
Oct 30 22:45:22.444203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.444177    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"]
Oct 30 22:45:22.448365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.448190    4581 scope.go:122] "RemoveContainer" containerID="546855bc56e82aa2bb719c57bf25920a92839bd2b46a09a151f06f7cf6ef377f"
Oct 30 22:45:22.469271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.468802    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890005-xmg85"]
Oct 30 22:45:22.476301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.476169    4581 scope.go:122] "RemoveContainer" containerID="fb8d3ae0654b0cbbbcc9a2b2fa2dbf54a754609d87d2965c8b9db797fcbc37e7"
Oct 30 22:45:22.537249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.536624    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"]
Oct 30 22:45:22.554813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.554779    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-54c7f0-29890005-mk7q8"]
Oct 30 22:45:22.560113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.560087    4581 scope.go:122] "RemoveContainer" containerID="e0494094f2ce291b277e2cbf7bcf545dac115b0ab5bee86a938f5e0214d4b543"
Oct 30 22:45:22.639366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.639339    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:22.641629 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.641601    4581 scope.go:122] "RemoveContainer" containerID="52ee624532f686244f779cbd0262edcf521de8d6f77954778a8321fd146e26d7"
Oct 30 22:45:22.642318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.642230    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:22.720197 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.717069    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"]
Oct 30 22:45:22.729596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.729569    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3acfdb-29890005-d6v4b"]
Oct 30 22:45:22.738625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.738595    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169c00be-1d02-4314-83bd-d27450715d23-kube-api-access-svzx4\" (UniqueName: \"kubernetes.io/projected/169c00be-1d02-4314-83bd-d27450715d23-kube-api-access-svzx4\") pod \"169c00be-1d02-4314-83bd-d27450715d23\" (UID: \"169c00be-1d02-4314-83bd-d27450715d23\") "
Oct 30 22:45:22.738776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.738760    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169c00be-1d02-4314-83bd-d27450715d23-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c00be-1d02-4314-83bd-d27450715d23-maas-api-service-ca\") pod \"169c00be-1d02-4314-83bd-d27450715d23\" (UID: \"169c00be-1d02-4314-83bd-d27450715d23\") "
Oct 30 22:45:22.739071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.739052    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd40171c-a1f5-484c-a078-ce72ef26cb90-kube-api-access-7kvmc\" (UniqueName: \"kubernetes.io/projected/fd40171c-a1f5-484c-a078-ce72ef26cb90-kube-api-access-7kvmc\") pod \"fd40171c-a1f5-484c-a078-ce72ef26cb90\" (UID: \"fd40171c-a1f5-484c-a078-ce72ef26cb90\") "
Oct 30 22:45:22.739286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.739209    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd40171c-a1f5-484c-a078-ce72ef26cb90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd40171c-a1f5-484c-a078-ce72ef26cb90-maas-api-service-ca\") pod \"fd40171c-a1f5-484c-a078-ce72ef26cb90\" (UID: \"fd40171c-a1f5-484c-a078-ce72ef26cb90\") "
Oct 30 22:45:22.741056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.740980    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd40171c-a1f5-484c-a078-ce72ef26cb90-maas-api-service-ca" pod "fd40171c-a1f5-484c-a078-ce72ef26cb90" (UID: "fd40171c-a1f5-484c-a078-ce72ef26cb90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.744919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.744888    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169c00be-1d02-4314-83bd-d27450715d23-kube-api-access-svzx4" pod "169c00be-1d02-4314-83bd-d27450715d23" (UID: "169c00be-1d02-4314-83bd-d27450715d23"). InnerVolumeSpecName "kube-api-access-svzx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.745380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.745196    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169c00be-1d02-4314-83bd-d27450715d23-maas-api-service-ca" pod "169c00be-1d02-4314-83bd-d27450715d23" (UID: "169c00be-1d02-4314-83bd-d27450715d23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.749042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.749015    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd40171c-a1f5-484c-a078-ce72ef26cb90-kube-api-access-7kvmc" pod "fd40171c-a1f5-484c-a078-ce72ef26cb90" (UID: "fd40171c-a1f5-484c-a078-ce72ef26cb90"). InnerVolumeSpecName "kube-api-access-7kvmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.783419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.783385    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:22.809326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.809304    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:22.814895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.814866    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:22.835683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.835647    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"]
Oct 30 22:45:22.841022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.840995    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb334b3a-3f29-4505-ae06-d4ce0917bebc-kube-api-access-wgf6g\" (UniqueName: \"kubernetes.io/projected/fb334b3a-3f29-4505-ae06-d4ce0917bebc-kube-api-access-wgf6g\") pod \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\" (UID: \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\") "
Oct 30 22:45:22.841196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.841179    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071439a8-bb97-4ab1-b4af-e0c88e7a585e-kube-api-access-cqbsx\" (UniqueName: \"kubernetes.io/projected/071439a8-bb97-4ab1-b4af-e0c88e7a585e-kube-api-access-cqbsx\") pod \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\" (UID: \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\") "
Oct 30 22:45:22.841461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.841444    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071439a8-bb97-4ab1-b4af-e0c88e7a585e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/071439a8-bb97-4ab1-b4af-e0c88e7a585e-maas-api-service-ca\") pod \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\" (UID: \"071439a8-bb97-4ab1-b4af-e0c88e7a585e\") "
Oct 30 22:45:22.841860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.841830    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071439a8-bb97-4ab1-b4af-e0c88e7a585e-maas-api-service-ca" pod "071439a8-bb97-4ab1-b4af-e0c88e7a585e" (UID: "071439a8-bb97-4ab1-b4af-e0c88e7a585e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.842003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.841978    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb334b3a-3f29-4505-ae06-d4ce0917bebc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb334b3a-3f29-4505-ae06-d4ce0917bebc-maas-api-service-ca\") pod \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\" (UID: \"fb334b3a-3f29-4505-ae06-d4ce0917bebc\") "
Oct 30 22:45:22.842592 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.842382    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb334b3a-3f29-4505-ae06-d4ce0917bebc-maas-api-service-ca" pod "fb334b3a-3f29-4505-ae06-d4ce0917bebc" (UID: "fb334b3a-3f29-4505-ae06-d4ce0917bebc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.843141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.843094    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvmc\" (UniqueName: \"kubernetes.io/projected/fd40171c-a1f5-484c-a078-ce72ef26cb90-kube-api-access-7kvmc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.843141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.843121    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd40171c-a1f5-484c-a078-ce72ef26cb90-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.843141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.843135    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svzx4\" (UniqueName: \"kubernetes.io/projected/169c00be-1d02-4314-83bd-d27450715d23-kube-api-access-svzx4\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.843454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.843149    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/071439a8-bb97-4ab1-b4af-e0c88e7a585e-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.843454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.843163    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c00be-1d02-4314-83bd-d27450715d23-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.843454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.843175    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb334b3a-3f29-4505-ae06-d4ce0917bebc-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.844507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.844197    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"]
Oct 30 22:45:22.845067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.845045    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb334b3a-3f29-4505-ae06-d4ce0917bebc-kube-api-access-wgf6g" pod "fb334b3a-3f29-4505-ae06-d4ce0917bebc" (UID: "fb334b3a-3f29-4505-ae06-d4ce0917bebc"). InnerVolumeSpecName "kube-api-access-wgf6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.845200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.845183    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071439a8-bb97-4ab1-b4af-e0c88e7a585e-kube-api-access-cqbsx" pod "071439a8-bb97-4ab1-b4af-e0c88e7a585e" (UID: "071439a8-bb97-4ab1-b4af-e0c88e7a585e"). InnerVolumeSpecName "kube-api-access-cqbsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.848904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.848860    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d85da4-29890005-qx4nq"]
Oct 30 22:45:22.944534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.943879    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f7488c-2538-4dc6-818b-306e295933cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f7488c-2538-4dc6-818b-306e295933cc-maas-api-service-ca\") pod \"a5f7488c-2538-4dc6-818b-306e295933cc\" (UID: \"a5f7488c-2538-4dc6-818b-306e295933cc\") "
Oct 30 22:45:22.944534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.943955    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f7488c-2538-4dc6-818b-306e295933cc-kube-api-access-j7276\" (UniqueName: \"kubernetes.io/projected/a5f7488c-2538-4dc6-818b-306e295933cc-kube-api-access-j7276\") pod \"a5f7488c-2538-4dc6-818b-306e295933cc\" (UID: \"a5f7488c-2538-4dc6-818b-306e295933cc\") "
Oct 30 22:45:22.944783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.944534    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgf6g\" (UniqueName: \"kubernetes.io/projected/fb334b3a-3f29-4505-ae06-d4ce0917bebc-kube-api-access-wgf6g\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.944783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.944551    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqbsx\" (UniqueName: \"kubernetes.io/projected/071439a8-bb97-4ab1-b4af-e0c88e7a585e-kube-api-access-cqbsx\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:22.953871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.952689    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f7488c-2538-4dc6-818b-306e295933cc-kube-api-access-j7276" pod "a5f7488c-2538-4dc6-818b-306e295933cc" (UID: "a5f7488c-2538-4dc6-818b-306e295933cc"). InnerVolumeSpecName "kube-api-access-j7276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.954261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:22.954233    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f7488c-2538-4dc6-818b-306e295933cc-maas-api-service-ca" pod "a5f7488c-2538-4dc6-818b-306e295933cc" (UID: "a5f7488c-2538-4dc6-818b-306e295933cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:23.072127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.063895    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f7488c-2538-4dc6-818b-306e295933cc-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:23.072127 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.063970    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7276\" (UniqueName: \"kubernetes.io/projected/a5f7488c-2538-4dc6-818b-306e295933cc-kube-api-access-j7276\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:23.132169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.132146    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:23.267042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.266960    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-kube-api-access-t8dz2\" (UniqueName: \"kubernetes.io/projected/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-kube-api-access-t8dz2\") pod \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\" (UID: \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\") "
Oct 30 22:45:23.267233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.267137    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-maas-api-service-ca\") pod \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\" (UID: \"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3\") "
Oct 30 22:45:23.267537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.267511    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-maas-api-service-ca" pod "71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" (UID: "71fc3c7a-b8bd-4e23-a857-09ffb342b6e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:23.268004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.267985    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:23.269300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.269271    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-kube-api-access-t8dz2" pod "71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" (UID: "71fc3c7a-b8bd-4e23-a857-09ffb342b6e3"). InnerVolumeSpecName "kube-api-access-t8dz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:23.342248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.342190    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"
Oct 30 22:45:23.342397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.342240    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw" event={"ID":"a5f7488c-2538-4dc6-818b-306e295933cc","Type":"ContainerDied","Data":"188a4b75c050d0d957bd12a03a7e88430a56347af1e4c39c0b47700663d493b7"}
Oct 30 22:45:23.342397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.342309    4581 scope.go:122] "RemoveContainer" containerID="4781ebd4c4f3da40fc799aa078451f96da14c1cd9333cf70cf9528418e67b870"
Oct 30 22:45:23.343999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.343980    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"
Oct 30 22:45:23.344123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.343980    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd" event={"ID":"fb334b3a-3f29-4505-ae06-d4ce0917bebc","Type":"ContainerDied","Data":"fd81750b24b1e087b8adbe21d001071efdfe980dab02ee789b72d28ec4e28a4b"}
Oct 30 22:45:23.358630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.358604    4581 scope.go:122] "RemoveContainer" containerID="8cc60b757662a3ac0968cc909b9013b2edddeae9d0845c0bdae5ecbfc83b6b04"
Oct 30 22:45:23.364575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.362399    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"
Oct 30 22:45:23.364575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.362422    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d" event={"ID":"fd40171c-a1f5-484c-a078-ce72ef26cb90","Type":"ContainerDied","Data":"ab2bd63c8740e35150b93e570a1ea025eedced838d1a6f35852cf2dcd283eedc"}
Oct 30 22:45:23.369887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.369858    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8dz2\" (UniqueName: \"kubernetes.io/projected/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3-kube-api-access-t8dz2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:23.370719 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.370697    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"
Oct 30 22:45:23.373374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.373333    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw" event={"ID":"71fc3c7a-b8bd-4e23-a857-09ffb342b6e3","Type":"ContainerDied","Data":"7c8705a16f48a86edd6166cceb527134628229137ef963c53f2276c8928a6836"}
Oct 30 22:45:23.386593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.386511    4581 scope.go:122] "RemoveContainer" containerID="d3894049e6014e14cd430474176200f991408d529723df14a15da35a7485af30"
Oct 30 22:45:23.389331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.388035    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm" event={"ID":"169c00be-1d02-4314-83bd-d27450715d23","Type":"ContainerDied","Data":"9345844b574c735f40694938bd6fcccf62e6b0b5cd03e24b1a92bd94dd16b394"}
Oct 30 22:45:23.389331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.388299    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"
Oct 30 22:45:23.400233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.400049    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc" event={"ID":"071439a8-bb97-4ab1-b4af-e0c88e7a585e","Type":"ContainerDied","Data":"ce5bb3635ae9b6ec789f6299ceb7d2e5b21d0a637201f4f6a5ff53459bdbe239"}
Oct 30 22:45:23.400233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.400137    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"
Oct 30 22:45:23.410585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.410564    4581 scope.go:122] "RemoveContainer" containerID="582f0c196b8cab5494e4fdbbdac541f3b7778fee9ab6b7a96e4a46ac77e07f19"
Oct 30 22:45:23.422480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.422436    4581 scope.go:122] "RemoveContainer" containerID="1f824563df1e677946143c7301d733ba8d010763653540e1afbc84ca3adcc6d0"
Oct 30 22:45:23.431336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.431283    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"]
Oct 30 22:45:23.440916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.440837    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890005-r4sgw"]
Oct 30 22:45:23.446782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.445396    4581 scope.go:122] "RemoveContainer" containerID="ca4373f3b4bc6ff85174a8ae1214711d33a3396499aa41c4ee84301d1d8826d7"
Oct 30 22:45:23.524987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.522464    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"]
Oct 30 22:45:23.528178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.528150    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0ad7c5-29890005-v9xcd"]
Oct 30 22:45:23.559152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.559121    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"]
Oct 30 22:45:23.559375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.559349    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-012ba0-29890005-ktpxm"]
Oct 30 22:45:23.587293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.587266    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:23.589478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.589456    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"]
Oct 30 22:45:23.595133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.595096    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-80ecb3-29890005-wzxjw"]
Oct 30 22:45:23.612379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.612353    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:23.624766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.624739    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"]
Oct 30 22:45:23.632036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.632011    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e0edc-29890005-sqgzc"]
Oct 30 22:45:23.657598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.657570    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"]
Oct 30 22:45:23.662573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.662545    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890005-kxz4d"]
Oct 30 22:45:23.673597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.673568    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-kube-api-access-qhfng\" (UniqueName: \"kubernetes.io/projected/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-kube-api-access-qhfng\") pod \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\" (UID: \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\") "
Oct 30 22:45:23.673759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.673743    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-kube-api-access-m5xtp\") pod \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\" (UID: \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\") "
Oct 30 22:45:23.673829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.673803    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-maas-api-service-ca\") pod \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\" (UID: \"dd3f411a-89df-4177-bb36-b9bdcca5e8c2\") "
Oct 30 22:45:23.673973 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.673956    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-maas-api-service-ca\") pod \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\" (UID: \"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb\") "
Oct 30 22:45:23.675744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.674558    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-maas-api-service-ca" pod "973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" (UID: "973aa012-ec8e-455f-9c5a-4fd55e3d4dbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:23.675744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.674573    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-maas-api-service-ca" pod "dd3f411a-89df-4177-bb36-b9bdcca5e8c2" (UID: "dd3f411a-89df-4177-bb36-b9bdcca5e8c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:23.676912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.676058    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:23.676912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.676089    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:23.684305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.677342    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-kube-api-access-qhfng" pod "dd3f411a-89df-4177-bb36-b9bdcca5e8c2" (UID: "dd3f411a-89df-4177-bb36-b9bdcca5e8c2"). InnerVolumeSpecName "kube-api-access-qhfng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:23.690266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.690209    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-kube-api-access-m5xtp" pod "973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" (UID: "973aa012-ec8e-455f-9c5a-4fd55e3d4dbb"). InnerVolumeSpecName "kube-api-access-m5xtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:23.778483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.778395    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhfng\" (UniqueName: \"kubernetes.io/projected/dd3f411a-89df-4177-bb36-b9bdcca5e8c2-kube-api-access-qhfng\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:23.778690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:23.778673    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb-kube-api-access-m5xtp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:24.106177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.106142    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" path="/var/lib/kubelet/pods/071439a8-bb97-4ab1-b4af-e0c88e7a585e/volumes"
Oct 30 22:45:24.106687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.106671    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169c00be-1d02-4314-83bd-d27450715d23" path="/var/lib/kubelet/pods/169c00be-1d02-4314-83bd-d27450715d23/volumes"
Oct 30 22:45:24.107082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.107069    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" path="/var/lib/kubelet/pods/2296d534-4a85-4b40-a54e-c8b2a67e2aa2/volumes"
Oct 30 22:45:24.107839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.107822    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" path="/var/lib/kubelet/pods/31f18846-74ba-48c9-a797-0f33d1820e16/volumes"
Oct 30 22:45:24.108265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.108252    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" path="/var/lib/kubelet/pods/71fc3c7a-b8bd-4e23-a857-09ffb342b6e3/volumes"
Oct 30 22:45:24.108661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.108648    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" path="/var/lib/kubelet/pods/8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3/volumes"
Oct 30 22:45:24.109348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.109333    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" path="/var/lib/kubelet/pods/a5f7488c-2538-4dc6-818b-306e295933cc/volumes"
Oct 30 22:45:24.109727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.109714    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfeb292-6e71-4543-9215-a445f8226c13" path="/var/lib/kubelet/pods/abfeb292-6e71-4543-9215-a445f8226c13/volumes"
Oct 30 22:45:24.110110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.110098    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" path="/var/lib/kubelet/pods/e93290c9-59d9-48e6-990a-b18d680913ed/volumes"
Oct 30 22:45:24.110801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.110786    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" path="/var/lib/kubelet/pods/fb334b3a-3f29-4505-ae06-d4ce0917bebc/volumes"
Oct 30 22:45:24.111167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.111155    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" path="/var/lib/kubelet/pods/fd40171c-a1f5-484c-a078-ce72ef26cb90/volumes"
Oct 30 22:45:24.418948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.418864    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"
Oct 30 22:45:24.419456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.418857    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v" event={"ID":"dd3f411a-89df-4177-bb36-b9bdcca5e8c2","Type":"ContainerDied","Data":"bccdf55a2ab4b23d940da03b9c7ed81ace0822f51f63db40d4e99cf52e64abd8"}
Oct 30 22:45:24.419456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.419016    4581 scope.go:122] "RemoveContainer" containerID="6cc8a555dda469569062d400d3c06f05f05e34d323aecc227c7aaf61331853d3"
Oct 30 22:45:24.424500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.424479    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"
Oct 30 22:45:24.424500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.424484    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc" event={"ID":"973aa012-ec8e-455f-9c5a-4fd55e3d4dbb","Type":"ContainerDied","Data":"a1f8f72e411a4a891c035ee7e936012e7f4705748c9ef9cf6deb7a3172e5651f"}
Oct 30 22:45:24.433116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.433072    4581 scope.go:122] "RemoveContainer" containerID="df793d5f88d43714879409824f5d6f0958ced0ddc18e5e772f2a9f93c64dfcc7"
Oct 30 22:45:24.469330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.469291    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"]
Oct 30 22:45:24.476481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.476446    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c13db0-29890005-5lp8v"]
Oct 30 22:45:24.502692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.502657    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"]
Oct 30 22:45:24.510200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:24.510170    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-40229e-29890005-55rsc"]
Oct 30 22:45:26.186242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:26.182366    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" path="/var/lib/kubelet/pods/973aa012-ec8e-455f-9c5a-4fd55e3d4dbb/volumes"
Oct 30 22:45:26.186242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:26.183110    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" path="/var/lib/kubelet/pods/dd3f411a-89df-4177-bb36-b9bdcca5e8c2/volumes"
Oct 30 22:45:40.822487 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:40.822443    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:45:40.846729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:40.846695    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"]
Oct 30 22:45:40.846959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:40.846916    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy" containerID="cri-o://3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412" gracePeriod=30
Oct 30 22:45:46.631149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.631125    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:45:46.678878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.678776    4581 generic.go:356] "Generic (PLEG): container finished" podID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerID="3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412" exitCode=0
Oct 30 22:45:46.678878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.678835    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"
Oct 30 22:45:46.678878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.678854    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" event={"ID":"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175","Type":"ContainerDied","Data":"3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412"}
Oct 30 22:45:46.679319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.678927    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n" event={"ID":"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175","Type":"ContainerDied","Data":"cc06e5b55b79db67aaa6f8841cdc7d8e634fd48aceae2c2f96896f0a026063ec"}
Oct 30 22:45:46.679319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.678955    4581 scope.go:122] "RemoveContainer" containerID="3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412"
Oct 30 22:45:46.692797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.692768    4581 scope.go:122] "RemoveContainer" containerID="2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712"
Oct 30 22:45:46.714366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.714336    4581 scope.go:122] "RemoveContainer" containerID="3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412"
Oct 30 22:45:46.714752 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:46.714728    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412\": container with ID starting with 3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412 not found: ID does not exist" containerID="3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412"
Oct 30 22:45:46.714840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.714763    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412"} err="failed to get container status \"3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412\": rpc error: code = NotFound desc = could not find container \"3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412\": container with ID starting with 3848edfa8da530d462e5162205cc959667a9b1c6603f371ee4fb72453e9d1412 not found: ID does not exist"
Oct 30 22:45:46.714840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.714782    4581 scope.go:122] "RemoveContainer" containerID="2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712"
Oct 30 22:45:46.715136 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:45:46.715113    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712\": container with ID starting with 2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712 not found: ID does not exist" containerID="2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712"
Oct 30 22:45:46.715267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.715142    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712"} err="failed to get container status \"2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712\": rpc error: code = NotFound desc = could not find container \"2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712\": container with ID starting with 2a25e365d0b18d1792d56a3e6353c0b61211ca016d148f9e8da03883e2eee712 not found: ID does not exist"
Oct 30 22:45:46.732790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.732765    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-socket\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.732955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.732828    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-kube-api-access-btmjd\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-kube-api-access-btmjd\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.732955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.732856    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-data\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.732955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.732942    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-envoy\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.733138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.733029    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-credential-socket\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.733138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.733057    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-socket" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:46.733138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.733094    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-certs\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.733159    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-podinfo\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.733317    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-token\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-token\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.733364    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istiod-ca-cert\") pod \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\" (UID: \"4c9c3de2-5c80-4f68-9ab3-9885ce1a6175\") "
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.733432    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-credential-socket" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.733802    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istiod-ca-cert" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.734023    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-certs" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.734164    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.734183    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.734197    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.734210    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:46.734780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.734280    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-data" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:46.735764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.735736    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-envoy" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:46.736299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.736266    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-kube-api-access-btmjd" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "kube-api-access-btmjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:46.736504 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.736481    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-token" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:46.746155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.746126    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-podinfo" pod "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" (UID: "4c9c3de2-5c80-4f68-9ab3-9885ce1a6175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:45:46.835844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.835817    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:46.835844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.835841    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:46.836064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.835853    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btmjd\" (UniqueName: \"kubernetes.io/projected/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-kube-api-access-btmjd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:46.836064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.835863    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:46.836064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:46.835872    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:47.024988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:47.024960    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"]
Oct 30 22:45:47.034066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:47.034039    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-qhw2n"]
Oct 30 22:45:48.099134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:48.099098    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" path="/var/lib/kubelet/pods/4c9c3de2-5c80-4f68-9ab3-9885ce1a6175/volumes"
Oct 30 22:45:49.691023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.690985    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"]
Oct 30 22:45:49.691767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691751    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy"
Oct 30 22:45:49.691816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691768    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy"
Oct 30 22:45:49.691816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691780    4581 cpu_manager.go:414] "containerMap: removing container" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:45:49.691816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691785    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:45:49.691816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691793    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:45:49.691816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691798    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:45:49.691816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691805    4581 cpu_manager.go:414] "containerMap: removing container" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:45:49.691816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691810    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691819    4581 cpu_manager.go:414] "containerMap: removing container" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691824    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691830    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691835    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691841    4581 cpu_manager.go:414] "containerMap: removing container" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691845    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691851    4581 cpu_manager.go:414] "containerMap: removing container" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691856    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691863    4581 cpu_manager.go:414] "containerMap: removing container" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691867    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691873    4581 cpu_manager.go:414] "containerMap: removing container" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691876    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691883    4581 cpu_manager.go:414] "containerMap: removing container" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691887    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691892    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691896    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691905    4581 cpu_manager.go:414] "containerMap: removing container" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691910    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691917    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691921    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691928    4581 cpu_manager.go:414] "containerMap: removing container" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691932    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691939    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691943    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691950    4581 cpu_manager.go:414] "containerMap: removing container" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691954    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691962    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691966    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691974    4581 cpu_manager.go:414] "containerMap: removing container" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691979    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691985    4581 cpu_manager.go:414] "containerMap: removing container" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691989    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.691997    4581 cpu_manager.go:414] "containerMap: removing container" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692000    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692008    4581 cpu_manager.go:414] "containerMap: removing container" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692012    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692018    4581 cpu_manager.go:414] "containerMap: removing container" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692023    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692031    4581 cpu_manager.go:414] "containerMap: removing container" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692035    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692041    4581 cpu_manager.go:414] "containerMap: removing container" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692045    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692052    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692056    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692062    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692066    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692074    4581 cpu_manager.go:414] "containerMap: removing container" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692078    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692084    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692089    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692096    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692100    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692106    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.692088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692110    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692117    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692122    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692127    4581 cpu_manager.go:414] "containerMap: removing container" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692131    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692140    4581 cpu_manager.go:414] "containerMap: removing container" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692144    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692150    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692154    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692162    4581 cpu_manager.go:414] "containerMap: removing container" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692166    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692177    4581 cpu_manager.go:414] "containerMap: removing container" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692181    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692193    4581 cpu_manager.go:414] "containerMap: removing container" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692198    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692204    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692208    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692231    4581 cpu_manager.go:414] "containerMap: removing container" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692235    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692240    4581 cpu_manager.go:414] "containerMap: removing container" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692244    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692251    4581 cpu_manager.go:414] "containerMap: removing container" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692255    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692261    4581 cpu_manager.go:414] "containerMap: removing container" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692265    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692375    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692384    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692389    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692395    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692402    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692407    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3bb4f9-d20b-4dc2-b0b9-702e9af23c2f" containerName="istio-proxy"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692414    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692419    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692426    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692435    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692440    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692445    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692451    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692458    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692463    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692471    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692477    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692481    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692487    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692491    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692496    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692501    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692507    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692512    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692516    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692523    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692529    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692535    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692541    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692545    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692550    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9c3de2-5c80-4f68-9ab3-9885ce1a6175" containerName="istio-proxy"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692556    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692562    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9af81a-12ae-4b6d-8d0a-8a90c3b11ba3" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692566    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692572    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fc3c7a-b8bd-4e23-a857-09ffb342b6e3" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692577    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692582    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692587    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692591    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692596    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692602    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692613    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692618    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692625    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692630    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2296d534-4a85-4b40-a54e-c8b2a67e2aa2" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692637    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:45:49.694183 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.692642    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd40171c-a1f5-484c-a078-ce72ef26cb90" containerName="cleanup"
Oct 30 22:45:49.696519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.696490    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.709891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.709867    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:49.719502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.719472    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:49.719648 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.719508    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:49.720707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.720673    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:45:49.727949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.727922    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"]
Oct 30 22:45:49.843684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.843651    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6g78\" (UniqueName: \"kubernetes.io/projected/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-kube-api-access-g6g78\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.843948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.843927    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.844045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.844029    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.945106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.945009    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.945106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.945072    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.945378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.945167    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6g78\" (UniqueName: \"kubernetes.io/projected/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-kube-api-access-g6g78\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.945742 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.945715    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.947655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.947628    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:49.962378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:49.962345    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6g78\" (UniqueName: \"kubernetes.io/projected/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-kube-api-access-g6g78\") pod \"maas-api-revoke-sub-be69375a-dxwts\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:50.021779 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:50.021746    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:50.336452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:50.336421    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"]
Oct 30 22:45:50.337620 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:45:50.337594    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d85666_683f_40bd_b9b8_c16f9a5a4cfe.slice/crio-4513d6f283ee7d081631ff6e95d1033869ace63bc251a22e542844539b704c29 WatchSource:0}: Error finding container 4513d6f283ee7d081631ff6e95d1033869ace63bc251a22e542844539b704c29: Status 404 returned error can't find the container with id 4513d6f283ee7d081631ff6e95d1033869ace63bc251a22e542844539b704c29
Oct 30 22:45:50.768034 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:50.767997    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts" event={"ID":"63d85666-683f-40bd-b9b8-c16f9a5a4cfe","Type":"ContainerStarted","Data":"4513d6f283ee7d081631ff6e95d1033869ace63bc251a22e542844539b704c29"}
Oct 30 22:45:51.781250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:51.781147    4581 generic.go:356] "Generic (PLEG): container finished" podID="63d85666-683f-40bd-b9b8-c16f9a5a4cfe" containerID="537ceb31f7e6b99dffcefedc4ccb088494d84da98b9b92356c0f5be2f488faa4" exitCode=0
Oct 30 22:45:51.781250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:51.781239    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts" event={"ID":"63d85666-683f-40bd-b9b8-c16f9a5a4cfe","Type":"ContainerDied","Data":"537ceb31f7e6b99dffcefedc4ccb088494d84da98b9b92356c0f5be2f488faa4"}
Oct 30 22:45:53.031456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.031414    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:53.143776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.143737    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-openshift-ca\") pod \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") "
Oct 30 22:45:53.143990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.143788    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-xks-ca\") pod \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") "
Oct 30 22:45:53.143990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.143927    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-kube-api-access-g6g78\" (UniqueName: \"kubernetes.io/projected/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-kube-api-access-g6g78\") pod \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\" (UID: \"63d85666-683f-40bd-b9b8-c16f9a5a4cfe\") "
Oct 30 22:45:53.144439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.144406    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-openshift-ca" pod "63d85666-683f-40bd-b9b8-c16f9a5a4cfe" (UID: "63d85666-683f-40bd-b9b8-c16f9a5a4cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:53.145162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.145140    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:53.147091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.147065    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-xks-ca" pod "63d85666-683f-40bd-b9b8-c16f9a5a4cfe" (UID: "63d85666-683f-40bd-b9b8-c16f9a5a4cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:53.147166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.147095    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-kube-api-access-g6g78" pod "63d85666-683f-40bd-b9b8-c16f9a5a4cfe" (UID: "63d85666-683f-40bd-b9b8-c16f9a5a4cfe"). InnerVolumeSpecName "kube-api-access-g6g78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:53.247586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.247548    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:53.247586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.247581    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6g78\" (UniqueName: \"kubernetes.io/projected/63d85666-683f-40bd-b9b8-c16f9a5a4cfe-kube-api-access-g6g78\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:45:53.804186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.804153    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"
Oct 30 22:45:53.804427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.804159    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts" event={"ID":"63d85666-683f-40bd-b9b8-c16f9a5a4cfe","Type":"ContainerDied","Data":"4513d6f283ee7d081631ff6e95d1033869ace63bc251a22e542844539b704c29"}
Oct 30 22:45:53.804427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:45:53.804314    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4513d6f283ee7d081631ff6e95d1033869ace63bc251a22e542844539b704c29"
Oct 30 22:46:01.173621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.173576    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"]
Oct 30 22:46:01.174568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174544    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:46:01.174568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174568    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f7488c-2538-4dc6-818b-306e295933cc" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174586    4581 cpu_manager.go:414] "containerMap: removing container" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174593    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174609    4581 cpu_manager.go:414] "containerMap: removing container" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174616    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174668    4581 cpu_manager.go:414] "containerMap: removing container" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174676    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169c00be-1d02-4314-83bd-d27450715d23" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174710    4581 cpu_manager.go:414] "containerMap: removing container" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174718    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174733    4581 cpu_manager.go:414] "containerMap: removing container" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174741    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636c8afd-b441-43f0-b5fb-ddc2a33dd2c9" containerName="istio-proxy"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174752    4581 cpu_manager.go:414] "containerMap: removing container" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:46:01.174763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174759    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174770    4581 cpu_manager.go:414] "containerMap: removing container" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174778    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decc6e06-9261-4213-a87b-26fb98bcbe50" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174801    4581 cpu_manager.go:414] "containerMap: removing container" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174808    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174821    4581 cpu_manager.go:414] "containerMap: removing container" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174828    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0420e-107e-40c7-8d92-85c50abb696d" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174839    4581 cpu_manager.go:414] "containerMap: removing container" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174847    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174857    4581 cpu_manager.go:414] "containerMap: removing container" podUID="63d85666-683f-40bd-b9b8-c16f9a5a4cfe" containerName="revoke-keys"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174865    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d85666-683f-40bd-b9b8-c16f9a5a4cfe" containerName="revoke-keys"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174884    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174892    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174903    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.174910    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175095    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="071439a8-bb97-4ab1-b4af-e0c88e7a585e" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175111    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93290c9-59d9-48e6-990a-b18d680913ed" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175123    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb334b3a-3f29-4505-ae06-d4ce0917bebc" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175131    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfeb292-6e71-4543-9215-a445f8226c13" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175140    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="973aa012-ec8e-455f-9c5a-4fd55e3d4dbb" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175150    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb7cbf-d1bd-4141-b0a1-6d7833377c53" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175160    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d85666-683f-40bd-b9b8-c16f9a5a4cfe" containerName="revoke-keys"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175168    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="abeab3fd-da86-4674-943d-0560abd2e752" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175177    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3f411a-89df-4177-bb36-b9bdcca5e8c2" containerName="cleanup"
Oct 30 22:46:01.175421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.175188    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f18846-74ba-48c9-a797-0f33d1820e16" containerName="cleanup"
Oct 30 22:46:01.181841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.181817    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.202538 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.202505    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"]
Oct 30 22:46:01.203497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.203472    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:01.203497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.203488    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:01.203702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.203476    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:01.203702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.203545    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:46:01.290327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.290281    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znslb\" (UniqueName: \"kubernetes.io/projected/80379a74-57ca-4d4e-87a7-0336960df0a6-kube-api-access-znslb\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.290526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.290399    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.290577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.290518    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.392308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.392274    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znslb\" (UniqueName: \"kubernetes.io/projected/80379a74-57ca-4d4e-87a7-0336960df0a6-kube-api-access-znslb\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.392500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.392353    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.392500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.392484    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.393165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.393088    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.394882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.394851    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.410165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.410135    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znslb\" (UniqueName: \"kubernetes.io/projected/80379a74-57ca-4d4e-87a7-0336960df0a6-kube-api-access-znslb\") pod \"maas-api-revoke-sub-fc4a3937-xcp5c\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.502944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.502854    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:01.820897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.820866    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"]
Oct 30 22:46:01.822019 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:46:01.821994    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80379a74_57ca_4d4e_87a7_0336960df0a6.slice/crio-54af4a9db505b302bf56d264fb0d83b75120783d76b78099ee3d844312a5b575 WatchSource:0}: Error finding container 54af4a9db505b302bf56d264fb0d83b75120783d76b78099ee3d844312a5b575: Status 404 returned error can't find the container with id 54af4a9db505b302bf56d264fb0d83b75120783d76b78099ee3d844312a5b575
Oct 30 22:46:01.893353 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:01.893316    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c" event={"ID":"80379a74-57ca-4d4e-87a7-0336960df0a6","Type":"ContainerStarted","Data":"54af4a9db505b302bf56d264fb0d83b75120783d76b78099ee3d844312a5b575"}
Oct 30 22:46:02.906409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:02.906373    4581 generic.go:356] "Generic (PLEG): container finished" podID="80379a74-57ca-4d4e-87a7-0336960df0a6" containerID="3ccefec74a77ca4e65b63e03fa85af09e7574095ce220082bfc96771ffabcaab" exitCode=0
Oct 30 22:46:02.906824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:02.906437    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c" event={"ID":"80379a74-57ca-4d4e-87a7-0336960df0a6","Type":"ContainerDied","Data":"3ccefec74a77ca4e65b63e03fa85af09e7574095ce220082bfc96771ffabcaab"}
Oct 30 22:46:04.134044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.134010    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:04.237606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.237514    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-xks-ca\") pod \"80379a74-57ca-4d4e-87a7-0336960df0a6\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") "
Oct 30 22:46:04.237606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.237560    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-openshift-ca\") pod \"80379a74-57ca-4d4e-87a7-0336960df0a6\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") "
Oct 30 22:46:04.237843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.237751    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80379a74-57ca-4d4e-87a7-0336960df0a6-kube-api-access-znslb\" (UniqueName: \"kubernetes.io/projected/80379a74-57ca-4d4e-87a7-0336960df0a6-kube-api-access-znslb\") pod \"80379a74-57ca-4d4e-87a7-0336960df0a6\" (UID: \"80379a74-57ca-4d4e-87a7-0336960df0a6\") "
Oct 30 22:46:04.237917 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.237833    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-openshift-ca" pod "80379a74-57ca-4d4e-87a7-0336960df0a6" (UID: "80379a74-57ca-4d4e-87a7-0336960df0a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:04.238446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.238420    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:04.239914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.239885    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80379a74-57ca-4d4e-87a7-0336960df0a6-kube-api-access-znslb" pod "80379a74-57ca-4d4e-87a7-0336960df0a6" (UID: "80379a74-57ca-4d4e-87a7-0336960df0a6"). InnerVolumeSpecName "kube-api-access-znslb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:04.240021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.239976    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-xks-ca" pod "80379a74-57ca-4d4e-87a7-0336960df0a6" (UID: "80379a74-57ca-4d4e-87a7-0336960df0a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:04.340498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.340464    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80379a74-57ca-4d4e-87a7-0336960df0a6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:04.340498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.340493    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znslb\" (UniqueName: \"kubernetes.io/projected/80379a74-57ca-4d4e-87a7-0336960df0a6-kube-api-access-znslb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:04.926009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.925979    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"
Oct 30 22:46:04.926192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.926024    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c" event={"ID":"80379a74-57ca-4d4e-87a7-0336960df0a6","Type":"ContainerDied","Data":"54af4a9db505b302bf56d264fb0d83b75120783d76b78099ee3d844312a5b575"}
Oct 30 22:46:04.926192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:04.926097    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54af4a9db505b302bf56d264fb0d83b75120783d76b78099ee3d844312a5b575"
Oct 30 22:46:10.819307 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:46:10.819265    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:46:19.722574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.722537    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"]
Oct 30 22:46:19.723346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.723328    4581 cpu_manager.go:414] "containerMap: removing container" podUID="80379a74-57ca-4d4e-87a7-0336960df0a6" containerName="revoke-keys"
Oct 30 22:46:19.723346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.723346    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80379a74-57ca-4d4e-87a7-0336960df0a6" containerName="revoke-keys"
Oct 30 22:46:19.723490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.723480    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="80379a74-57ca-4d4e-87a7-0336960df0a6" containerName="revoke-keys"
Oct 30 22:46:19.727657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.727642    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.746239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.746196    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlv49\" (UniqueName: \"kubernetes.io/projected/46216be0-7d8d-4f70-95cf-af856b684f7b-kube-api-access-qlv49\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.746537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.746519    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.746902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.746556    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.750073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.750034    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:19.750195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.750083    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:19.750195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.750180    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:19.750351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.750318    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:46:19.770036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.769952    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"]
Oct 30 22:46:19.848016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.847975    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.848016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.848020    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.848345 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.848120    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlv49\" (UniqueName: \"kubernetes.io/projected/46216be0-7d8d-4f70-95cf-af856b684f7b-kube-api-access-qlv49\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.848774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.848751    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.850582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.850561    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:19.865868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:19.865834    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlv49\" (UniqueName: \"kubernetes.io/projected/46216be0-7d8d-4f70-95cf-af856b684f7b-kube-api-access-qlv49\") pod \"maas-api-revoke-keys-e2e-inf-a-999470-pxt22\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:20.042690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:20.042606    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:20.364508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:20.364477    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"]
Oct 30 22:46:20.366425 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:46:20.366391    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46216be0_7d8d_4f70_95cf_af856b684f7b.slice/crio-59f7d94c12002985a376881586882e53d1fefa3cc5af8716995917b4450c9eaa WatchSource:0}: Error finding container 59f7d94c12002985a376881586882e53d1fefa3cc5af8716995917b4450c9eaa: Status 404 returned error can't find the container with id 59f7d94c12002985a376881586882e53d1fefa3cc5af8716995917b4450c9eaa
Oct 30 22:46:21.119350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:21.119307    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22" event={"ID":"46216be0-7d8d-4f70-95cf-af856b684f7b","Type":"ContainerStarted","Data":"a1f7fd00797673afbc889355506794665c06a895b20453b6248239899c6f8aed"}
Oct 30 22:46:21.119861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:21.119375    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22" event={"ID":"46216be0-7d8d-4f70-95cf-af856b684f7b","Type":"ContainerStarted","Data":"59f7d94c12002985a376881586882e53d1fefa3cc5af8716995917b4450c9eaa"}
Oct 30 22:46:21.159484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:21.159368    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22" podStartSLOduration=2.159348248 podStartE2EDuration="2.159348248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:21.155846166 +0000 UTC m=+2365.194785053" watchObservedRunningTime="2026-10-30 22:46:21.159348248 +0000 UTC m=+2365.198287135"
Oct 30 22:46:22.132670 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:22.132624    4581 generic.go:356] "Generic (PLEG): container finished" podID="46216be0-7d8d-4f70-95cf-af856b684f7b" containerID="a1f7fd00797673afbc889355506794665c06a895b20453b6248239899c6f8aed" exitCode=0
Oct 30 22:46:22.133254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:22.132763    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22" event={"ID":"46216be0-7d8d-4f70-95cf-af856b684f7b","Type":"ContainerDied","Data":"a1f7fd00797673afbc889355506794665c06a895b20453b6248239899c6f8aed"}
Oct 30 22:46:23.418272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.418178    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:23.467053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.467006    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46216be0-7d8d-4f70-95cf-af856b684f7b-kube-api-access-qlv49\" (UniqueName: \"kubernetes.io/projected/46216be0-7d8d-4f70-95cf-af856b684f7b-kube-api-access-qlv49\") pod \"46216be0-7d8d-4f70-95cf-af856b684f7b\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") "
Oct 30 22:46:23.468403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.468372    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-xks-ca\") pod \"46216be0-7d8d-4f70-95cf-af856b684f7b\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") "
Oct 30 22:46:23.468638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.468621    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-openshift-ca\") pod \"46216be0-7d8d-4f70-95cf-af856b684f7b\" (UID: \"46216be0-7d8d-4f70-95cf-af856b684f7b\") "
Oct 30 22:46:23.469667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.469639    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-openshift-ca" pod "46216be0-7d8d-4f70-95cf-af856b684f7b" (UID: "46216be0-7d8d-4f70-95cf-af856b684f7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:23.474937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.474890    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-xks-ca" pod "46216be0-7d8d-4f70-95cf-af856b684f7b" (UID: "46216be0-7d8d-4f70-95cf-af856b684f7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:23.481380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.481352    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46216be0-7d8d-4f70-95cf-af856b684f7b-kube-api-access-qlv49" pod "46216be0-7d8d-4f70-95cf-af856b684f7b" (UID: "46216be0-7d8d-4f70-95cf-af856b684f7b"). InnerVolumeSpecName "kube-api-access-qlv49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:23.571286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.571252    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:23.571286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.571285    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlv49\" (UniqueName: \"kubernetes.io/projected/46216be0-7d8d-4f70-95cf-af856b684f7b-kube-api-access-qlv49\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:23.571493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:23.571300    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46216be0-7d8d-4f70-95cf-af856b684f7b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:24.157611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:24.157575    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22" event={"ID":"46216be0-7d8d-4f70-95cf-af856b684f7b","Type":"ContainerDied","Data":"59f7d94c12002985a376881586882e53d1fefa3cc5af8716995917b4450c9eaa"}
Oct 30 22:46:24.157611 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:24.157598    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"
Oct 30 22:46:24.157880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:24.157636    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59f7d94c12002985a376881586882e53d1fefa3cc5af8716995917b4450c9eaa"
Oct 30 22:46:27.246836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:27.246798    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"]
Oct 30 22:46:27.247366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:27.247057    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy" containerID="cri-o://d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493" gracePeriod=30
Oct 30 22:46:30.316623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:30.316590    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"]
Oct 30 22:46:30.350281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:30.350247    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"]
Oct 30 22:46:32.557441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.557411    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:46:32.683979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.683942    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-kube-api-access-64m2w\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-kube-api-access-64m2w\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684001    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-socket\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684186    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-envoy\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684249    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-podinfo\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684313    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-credential-socket\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684376    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-certs\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684390    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-socket" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:32.684469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684427    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-token\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-token\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684503    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-data\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684633    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e082c55a-1402-4954-a044-f1bf8d75bfee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c55a-1402-4954-a044-f1bf8d75bfee-istiod-ca-cert\") pod \"e082c55a-1402-4954-a044-f1bf8d75bfee\" (UID: \"e082c55a-1402-4954-a044-f1bf8d75bfee\") "
Oct 30 22:46:32.684802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684768    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-certs" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:32.684879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.684833    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-credential-socket" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:32.685166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.685134    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-data" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:32.685316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.685187    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e082c55a-1402-4954-a044-f1bf8d75bfee-istiod-ca-cert" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:32.685877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.685851    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:32.685978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.685885    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:32.685978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.685903    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:32.685978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.685915    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e082c55a-1402-4954-a044-f1bf8d75bfee-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:32.685978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.685930    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:32.686790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.686763    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-envoy" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:32.686983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.686953    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-podinfo" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:46:32.688082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.688050    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-token" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:32.688153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.688087    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-kube-api-access-64m2w" pod "e082c55a-1402-4954-a044-f1bf8d75bfee" (UID: "e082c55a-1402-4954-a044-f1bf8d75bfee"). InnerVolumeSpecName "kube-api-access-64m2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:32.787609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.787511    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:32.787609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.787545    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:32.787609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.787557    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:32.787609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:32.787570    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64m2w\" (UniqueName: \"kubernetes.io/projected/e082c55a-1402-4954-a044-f1bf8d75bfee-kube-api-access-64m2w\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:33.263148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.263114    4581 generic.go:356] "Generic (PLEG): container finished" podID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerID="d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493" exitCode=0
Oct 30 22:46:33.263411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.263168    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"
Oct 30 22:46:33.263411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.263248    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" event={"ID":"e082c55a-1402-4954-a044-f1bf8d75bfee","Type":"ContainerDied","Data":"d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493"}
Oct 30 22:46:33.263411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.263313    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7" event={"ID":"e082c55a-1402-4954-a044-f1bf8d75bfee","Type":"ContainerDied","Data":"e6abb4f884fde58bbec8f8aff0d3e014013701bdd2eba0fa2668e632dc89afa0"}
Oct 30 22:46:33.263411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.263342    4581 scope.go:122] "RemoveContainer" containerID="d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493"
Oct 30 22:46:33.275963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.275944    4581 scope.go:122] "RemoveContainer" containerID="d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493"
Oct 30 22:46:33.276289 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:46:33.276246    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493\": container with ID starting with d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493 not found: ID does not exist" containerID="d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493"
Oct 30 22:46:33.276349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.276301    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493"} err="failed to get container status \"d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493\": rpc error: code = NotFound desc = could not find container \"d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493\": container with ID starting with d22726cb6df9d2e55dcb0507fab0d5f8c5e408c1adec9e3cb3b8161437257493 not found: ID does not exist"
Oct 30 22:46:33.321076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.321048    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"]
Oct 30 22:46:33.325866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:33.325838    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-47dz7"]
Oct 30 22:46:34.098937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:34.098904    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" path="/var/lib/kubelet/pods/e082c55a-1402-4954-a044-f1bf8d75bfee/volumes"
Oct 30 22:46:35.317404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:35.317358    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956" podUID="b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" containerName="payload-processing" containerID="cri-o://76bce2d210285b28c7e1d8d81c95971983ae433c8fd24abb9e57178d0111592d" gracePeriod=25
Oct 30 22:46:35.351334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:35.351299    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm" podUID="df98ade7-2aa1-4bea-b2f2-6901e7564e5a" containerName="payload-pre-processing" containerID="cri-o://a4414354e6b1a3c46a0c873dd66faee563c7e7dfdc1dcfa317a1c868b6178ed0" gracePeriod=25
Oct 30 22:46:39.037780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:39.037689    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956" podUID="b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:39.038182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:39.037903    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm" podUID="df98ade7-2aa1-4bea-b2f2-6901e7564e5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:40.819146 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:46:40.819097    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:46:40.981432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:40.981399    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"]
Oct 30 22:46:40.981654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:40.981628    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" containerID="cri-o://1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a" gracePeriod=30
Oct 30 22:46:44.046790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:44.046758    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"]
Oct 30 22:46:44.054369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:44.054331    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ebb966-7vm5d"]
Oct 30 22:46:44.102107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:44.102076    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c3e2bb-13ff-4296-9b52-24d50d414b6c" path="/var/lib/kubelet/pods/e7c3e2bb-13ff-4296-9b52-24d50d414b6c/volumes"
Oct 30 22:46:44.809201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:44.809150    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sv4fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:46:44.809609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:44.809317    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:46:46.310007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.309982    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:46:46.342007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.341976    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-envoy\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342033    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-certs\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342054    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-data\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342080    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75e7f902-6226-48c4-a92e-b566be595f39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e7f902-6226-48c4-a92e-b566be595f39-istio-podinfo\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342156    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e7f902-6226-48c4-a92e-b566be595f39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e7f902-6226-48c4-a92e-b566be595f39-istiod-ca-cert\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342231    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-kube-api-access-n4gcv\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-kube-api-access-n4gcv\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342355    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-credential-socket\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342342    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-certs" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:46.342475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342409    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-data" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:46.342475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342455    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-istio-token\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-istio-token\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342505    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-socket\") pod \"75e7f902-6226-48c4-a92e-b566be595f39\" (UID: \"75e7f902-6226-48c4-a92e-b566be595f39\") "
Oct 30 22:46:46.342904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342694    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-credential-socket" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:46.342904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342851    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-socket" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:46.342904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.342890    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e7f902-6226-48c4-a92e-b566be595f39-istiod-ca-cert" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:46.343518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.343493    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e7f902-6226-48c4-a92e-b566be595f39-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.343636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.343522    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.343636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.343540    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.343636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.343555    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.343636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.343569    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.344574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.344545    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-envoy" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:46.344885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.344864    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75e7f902-6226-48c4-a92e-b566be595f39-istio-podinfo" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:46:46.345318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.345296    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-kube-api-access-n4gcv" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "kube-api-access-n4gcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:46.345396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.345350    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-istio-token" pod "75e7f902-6226-48c4-a92e-b566be595f39" (UID: "75e7f902-6226-48c4-a92e-b566be595f39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:46.401237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.401185    4581 generic.go:356] "Generic (PLEG): container finished" podID="75e7f902-6226-48c4-a92e-b566be595f39" containerID="1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a" exitCode=0
Oct 30 22:46:46.401429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.401266    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" event={"ID":"75e7f902-6226-48c4-a92e-b566be595f39","Type":"ContainerDied","Data":"1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a"}
Oct 30 22:46:46.401429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.401295    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"
Oct 30 22:46:46.401429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.401339    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp" event={"ID":"75e7f902-6226-48c4-a92e-b566be595f39","Type":"ContainerDied","Data":"91b2e35091ce519904f504a74030d97067560d89d8b6b8115baa10cf386fd07f"}
Oct 30 22:46:46.401429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.401389    4581 scope.go:122] "RemoveContainer" containerID="1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a"
Oct 30 22:46:46.418589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.418553    4581 scope.go:122] "RemoveContainer" containerID="1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a"
Oct 30 22:46:46.419044 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:46:46.418973    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a\": container with ID starting with 1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a not found: ID does not exist" containerID="1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a"
Oct 30 22:46:46.419044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.419016    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a"} err="failed to get container status \"1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a\": rpc error: code = NotFound desc = could not find container \"1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a\": container with ID starting with 1072d2398dd219b81fbbd162b0452715c71382f0d0c82123e7a0c816a20ea64a not found: ID does not exist"
Oct 30 22:46:46.445805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.445771    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4gcv\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-kube-api-access-n4gcv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.445805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.445804    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e7f902-6226-48c4-a92e-b566be595f39-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.445805 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.445814    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e7f902-6226-48c4-a92e-b566be595f39-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.446121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.445822    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e7f902-6226-48c4-a92e-b566be595f39-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:46.452331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.452298    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"]
Oct 30 22:46:46.455173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:46.455144    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sv4fp"]
Oct 30 22:46:48.101759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.101724    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e7f902-6226-48c4-a92e-b566be595f39" path="/var/lib/kubelet/pods/75e7f902-6226-48c4-a92e-b566be595f39/volumes"
Oct 30 22:46:48.684146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.684105    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"]
Oct 30 22:46:48.685265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685245    4581 cpu_manager.go:414] "containerMap: removing container" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy"
Oct 30 22:46:48.685339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685269    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy"
Oct 30 22:46:48.685339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685311    4581 cpu_manager.go:414] "containerMap: removing container" podUID="46216be0-7d8d-4f70-95cf-af856b684f7b" containerName="revoke-keys"
Oct 30 22:46:48.685339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685319    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46216be0-7d8d-4f70-95cf-af856b684f7b" containerName="revoke-keys"
Oct 30 22:46:48.685439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685344    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy"
Oct 30 22:46:48.685439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685352    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy"
Oct 30 22:46:48.685572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685560    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e7f902-6226-48c4-a92e-b566be595f39" containerName="istio-proxy"
Oct 30 22:46:48.685608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685577    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="46216be0-7d8d-4f70-95cf-af856b684f7b" containerName="revoke-keys"
Oct 30 22:46:48.685608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.685592    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e082c55a-1402-4954-a044-f1bf8d75bfee" containerName="istio-proxy"
Oct 30 22:46:48.691637 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.691612    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.708050 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.708017    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"]
Oct 30 22:46:48.709485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.709456    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:48.709752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.709723    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:48.715268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.713747    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:48.715268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.713858    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:46:48.783357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.783321    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"]
Oct 30 22:46:48.788108 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.788084    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:48.798240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.795590    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vgk\" (UniqueName: \"kubernetes.io/projected/5d3c5bee-4af9-4346-b452-f1501a45b0b6-kube-api-access-v4vgk\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.798240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.795704    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.798240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.795943    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.811097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.811059    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"]
Oct 30 22:46:48.897505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.897470    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4tfr\" (UniqueName: \"kubernetes.io/projected/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-kube-api-access-k4tfr\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:48.897714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.897546    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.897714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.897664    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:48.897847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.897829    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vgk\" (UniqueName: \"kubernetes.io/projected/5d3c5bee-4af9-4346-b452-f1501a45b0b6-kube-api-access-v4vgk\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.897944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.897926    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.898031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.898012    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:48.898421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.898395    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.900616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.900582    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.914788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.914767    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vgk\" (UniqueName: \"kubernetes.io/projected/5d3c5bee-4af9-4346-b452-f1501a45b0b6-kube-api-access-v4vgk\") pod \"maas-api-revoke-sub-91ccb405-n954h\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:48.999741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.999650    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:48.999908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.999758    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4tfr\" (UniqueName: \"kubernetes.io/projected/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-kube-api-access-k4tfr\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:48.999908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:48.999819    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:49.000539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.000512    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:49.002467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.002442    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:49.009527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.009507    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:49.016897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.016876    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4tfr\" (UniqueName: \"kubernetes.io/projected/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-kube-api-access-k4tfr\") pod \"maas-api-revoke-sub-284bc491-mtvcm\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:49.038111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.038073    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm" podUID="df98ade7-2aa1-4bea-b2f2-6901e7564e5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:49.038352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.038328    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956" podUID="b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:49.110020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.109992    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:49.322565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.322538    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"]
Oct 30 22:46:49.323880 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:46:49.323848    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3c5bee_4af9_4346_b452_f1501a45b0b6.slice/crio-4571cb1c36f2f4c3073f319973184d290e3f075ac9f8f991c37c1df91b4d61cb WatchSource:0}: Error finding container 4571cb1c36f2f4c3073f319973184d290e3f075ac9f8f991c37c1df91b4d61cb: Status 404 returned error can't find the container with id 4571cb1c36f2f4c3073f319973184d290e3f075ac9f8f991c37c1df91b4d61cb
Oct 30 22:46:49.433867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.433825    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h" event={"ID":"5d3c5bee-4af9-4346-b452-f1501a45b0b6","Type":"ContainerStarted","Data":"4571cb1c36f2f4c3073f319973184d290e3f075ac9f8f991c37c1df91b4d61cb"}
Oct 30 22:46:49.442101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.442073    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"]
Oct 30 22:46:49.442628 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:46:49.442600    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21ea171_bbf6_47f1_ab4e_f7bd11e3cc29.slice/crio-c89d38ec1db4ccaf3e26c993a57fb00ac3f6b43d768f74d9312b4db873983687 WatchSource:0}: Error finding container c89d38ec1db4ccaf3e26c993a57fb00ac3f6b43d768f74d9312b4db873983687: Status 404 returned error can't find the container with id c89d38ec1db4ccaf3e26c993a57fb00ac3f6b43d768f74d9312b4db873983687
Oct 30 22:46:49.802152 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.802119    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"]
Oct 30 22:46:49.806815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.806796    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:49.820789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.820755    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b03b33\""
Oct 30 22:46:49.823525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.823502    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"]
Oct 30 22:46:49.872160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.868372    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"]
Oct 30 22:46:49.874332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.874306    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:49.892727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.890757    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"]
Oct 30 22:46:49.936245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.927466    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b03b33-dockercfg-6ggwr\""
Oct 30 22:46:49.978252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.961855    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwns\" (UniqueName: \"kubernetes.io/projected/257cfb11-d754-4a1b-9923-73dedd664bfd-kube-api-access-tnwns\") pod \"payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v\" (UID: \"257cfb11-d754-4a1b-9923-73dedd664bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:49.978252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.961934    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1681e335-1250-413f-bc87-61ea076bd6d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76\" (UID: \"1681e335-1250-413f-bc87-61ea076bd6d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:49.978252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.962065    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8kg\" (UniqueName: \"kubernetes.io/projected/1681e335-1250-413f-bc87-61ea076bd6d9-kube-api-access-nr8kg\") pod \"payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76\" (UID: \"1681e335-1250-413f-bc87-61ea076bd6d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:49.978252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:49.962106    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257cfb11-d754-4a1b-9923-73dedd664bfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v\" (UID: \"257cfb11-d754-4a1b-9923-73dedd664bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:50.076723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.066135    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwns\" (UniqueName: \"kubernetes.io/projected/257cfb11-d754-4a1b-9923-73dedd664bfd-kube-api-access-tnwns\") pod \"payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v\" (UID: \"257cfb11-d754-4a1b-9923-73dedd664bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:50.076723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.066202    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1681e335-1250-413f-bc87-61ea076bd6d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76\" (UID: \"1681e335-1250-413f-bc87-61ea076bd6d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:50.076723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.066353    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8kg\" (UniqueName: \"kubernetes.io/projected/1681e335-1250-413f-bc87-61ea076bd6d9-kube-api-access-nr8kg\") pod \"payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76\" (UID: \"1681e335-1250-413f-bc87-61ea076bd6d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:50.076723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.066388    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257cfb11-d754-4a1b-9923-73dedd664bfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v\" (UID: \"257cfb11-d754-4a1b-9923-73dedd664bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:50.076723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.067480    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257cfb11-d754-4a1b-9923-73dedd664bfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v\" (UID: \"257cfb11-d754-4a1b-9923-73dedd664bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:50.076723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.068648    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1681e335-1250-413f-bc87-61ea076bd6d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76\" (UID: \"1681e335-1250-413f-bc87-61ea076bd6d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:50.227712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.226308    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwns\" (UniqueName: \"kubernetes.io/projected/257cfb11-d754-4a1b-9923-73dedd664bfd-kube-api-access-tnwns\") pod \"payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v\" (UID: \"257cfb11-d754-4a1b-9923-73dedd664bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:50.266263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.265957    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8kg\" (UniqueName: \"kubernetes.io/projected/1681e335-1250-413f-bc87-61ea076bd6d9-kube-api-access-nr8kg\") pod \"payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76\" (UID: \"1681e335-1250-413f-bc87-61ea076bd6d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:50.422605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.422573    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:50.446575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.446538    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm" event={"ID":"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29","Type":"ContainerStarted","Data":"f4e3c890d908390e375a474ab6a5ce538abd5159aac3a98083e2138735e47055"}
Oct 30 22:46:50.446769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.446600    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm" event={"ID":"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29","Type":"ContainerStarted","Data":"c89d38ec1db4ccaf3e26c993a57fb00ac3f6b43d768f74d9312b4db873983687"}
Oct 30 22:46:50.448553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.448522    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h" event={"ID":"5d3c5bee-4af9-4346-b452-f1501a45b0b6","Type":"ContainerStarted","Data":"d2740b1fdfa69cb227705b2cc9196e6e14f54716e9da0a3098e233c9582f65ac"}
Oct 30 22:46:50.489143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.488089    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm" podStartSLOduration=2.488067093 podStartE2EDuration="2.488067093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:50.482080397 +0000 UTC m=+2394.521019317" watchObservedRunningTime="2026-10-30 22:46:50.488067093 +0000 UTC m=+2394.527005980"
Oct 30 22:46:50.495315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.495276    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:50.525965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.525857    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h" podStartSLOduration=2.525837119 podStartE2EDuration="2.525837119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:50.516775796 +0000 UTC m=+2394.555714684" watchObservedRunningTime="2026-10-30 22:46:50.525837119 +0000 UTC m=+2394.564776008"
Oct 30 22:46:50.744957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.744924    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"]
Oct 30 22:46:50.745207 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:46:50.745038    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257cfb11_d754_4a1b_9923_73dedd664bfd.slice/crio-748dd2ed457a8e778e2c2cc33232e9066e0621ad404332dabe9cc77b105efa50 WatchSource:0}: Error finding container 748dd2ed457a8e778e2c2cc33232e9066e0621ad404332dabe9cc77b105efa50: Status 404 returned error can't find the container with id 748dd2ed457a8e778e2c2cc33232e9066e0621ad404332dabe9cc77b105efa50
Oct 30 22:46:50.834979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:50.834953    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"]
Oct 30 22:46:50.837160 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:46:50.837127    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1681e335_1250_413f_bc87_61ea076bd6d9.slice/crio-d8574eaf652aa1ffd25702de3a7b99d96bf04c283f04e7b6d82354e30a2100c5 WatchSource:0}: Error finding container d8574eaf652aa1ffd25702de3a7b99d96bf04c283f04e7b6d82354e30a2100c5: Status 404 returned error can't find the container with id d8574eaf652aa1ffd25702de3a7b99d96bf04c283f04e7b6d82354e30a2100c5
Oct 30 22:46:51.460431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.460390    4581 generic.go:356] "Generic (PLEG): container finished" podID="c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29" containerID="f4e3c890d908390e375a474ab6a5ce538abd5159aac3a98083e2138735e47055" exitCode=0
Oct 30 22:46:51.460911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.460472    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm" event={"ID":"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29","Type":"ContainerDied","Data":"f4e3c890d908390e375a474ab6a5ce538abd5159aac3a98083e2138735e47055"}
Oct 30 22:46:51.462410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.462385    4581 generic.go:356] "Generic (PLEG): container finished" podID="5d3c5bee-4af9-4346-b452-f1501a45b0b6" containerID="d2740b1fdfa69cb227705b2cc9196e6e14f54716e9da0a3098e233c9582f65ac" exitCode=0
Oct 30 22:46:51.462559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.462457    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h" event={"ID":"5d3c5bee-4af9-4346-b452-f1501a45b0b6","Type":"ContainerDied","Data":"d2740b1fdfa69cb227705b2cc9196e6e14f54716e9da0a3098e233c9582f65ac"}
Oct 30 22:46:51.464203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.464180    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76" event={"ID":"1681e335-1250-413f-bc87-61ea076bd6d9","Type":"ContainerStarted","Data":"6c481f3482e4995191f2bf65b344a845b8d512e485185789b52b898613b93d5c"}
Oct 30 22:46:51.464341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.464258    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76" event={"ID":"1681e335-1250-413f-bc87-61ea076bd6d9","Type":"ContainerStarted","Data":"d8574eaf652aa1ffd25702de3a7b99d96bf04c283f04e7b6d82354e30a2100c5"}
Oct 30 22:46:51.464341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.464289    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:46:51.466165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.466141    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v" event={"ID":"257cfb11-d754-4a1b-9923-73dedd664bfd","Type":"ContainerStarted","Data":"2333cbcedd71f6ae32ef963e2e329adb018c16714e4c218d43a1bd796938b5cd"}
Oct 30 22:46:51.466295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.466184    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v" event={"ID":"257cfb11-d754-4a1b-9923-73dedd664bfd","Type":"ContainerStarted","Data":"748dd2ed457a8e778e2c2cc33232e9066e0621ad404332dabe9cc77b105efa50"}
Oct 30 22:46:51.466295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.466279    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:46:51.566404 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.566335    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76" podStartSLOduration=2.566315331 podStartE2EDuration="2.566315331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:51.561533106 +0000 UTC m=+2395.600471994" watchObservedRunningTime="2026-10-30 22:46:51.566315331 +0000 UTC m=+2395.605254217"
Oct 30 22:46:51.603414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:51.603360    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v" podStartSLOduration=2.603343739 podStartE2EDuration="2.603343739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:51.594319802 +0000 UTC m=+2395.633258691" watchObservedRunningTime="2026-10-30 22:46:51.603343739 +0000 UTC m=+2395.642282630"
Oct 30 22:46:52.669527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.669496    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:52.739801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.739720    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:52.748046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.748012    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-kube-api-access-k4tfr\" (UniqueName: \"kubernetes.io/projected/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-kube-api-access-k4tfr\") pod \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") "
Oct 30 22:46:52.748302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.748253    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-xks-ca\") pod \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") "
Oct 30 22:46:52.748494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.748458    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-openshift-ca\") pod \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\" (UID: \"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29\") "
Oct 30 22:46:52.749484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.749454    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-openshift-ca" pod "c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29" (UID: "c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:52.751696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.751657    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-kube-api-access-k4tfr" pod "c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29" (UID: "c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29"). InnerVolumeSpecName "kube-api-access-k4tfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:52.754005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.753976    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-xks-ca" pod "c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29" (UID: "c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:52.850634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.850602    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-xks-ca\") pod \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") "
Oct 30 22:46:52.850822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.850776    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3c5bee-4af9-4346-b452-f1501a45b0b6-kube-api-access-v4vgk\" (UniqueName: \"kubernetes.io/projected/5d3c5bee-4af9-4346-b452-f1501a45b0b6-kube-api-access-v4vgk\") pod \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") "
Oct 30 22:46:52.850822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.850813    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-openshift-ca\") pod \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\" (UID: \"5d3c5bee-4af9-4346-b452-f1501a45b0b6\") "
Oct 30 22:46:52.851244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.851185    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-openshift-ca" pod "5d3c5bee-4af9-4346-b452-f1501a45b0b6" (UID: "5d3c5bee-4af9-4346-b452-f1501a45b0b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:52.851762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.851737    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:52.851850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.851766    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4tfr\" (UniqueName: \"kubernetes.io/projected/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-kube-api-access-k4tfr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:52.851850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.851782    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:52.851850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.851797    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:52.852935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.852902    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-xks-ca" pod "5d3c5bee-4af9-4346-b452-f1501a45b0b6" (UID: "5d3c5bee-4af9-4346-b452-f1501a45b0b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:52.853022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.853005    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3c5bee-4af9-4346-b452-f1501a45b0b6-kube-api-access-v4vgk" pod "5d3c5bee-4af9-4346-b452-f1501a45b0b6" (UID: "5d3c5bee-4af9-4346-b452-f1501a45b0b6"). InnerVolumeSpecName "kube-api-access-v4vgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:52.953468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.953432    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3c5bee-4af9-4346-b452-f1501a45b0b6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:52.953468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:52.953462    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vgk\" (UniqueName: \"kubernetes.io/projected/5d3c5bee-4af9-4346-b452-f1501a45b0b6-kube-api-access-v4vgk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:53.489812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:53.489777    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"
Oct 30 22:46:53.490031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:53.489809    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm" event={"ID":"c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29","Type":"ContainerDied","Data":"c89d38ec1db4ccaf3e26c993a57fb00ac3f6b43d768f74d9312b4db873983687"}
Oct 30 22:46:53.490031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:53.489868    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89d38ec1db4ccaf3e26c993a57fb00ac3f6b43d768f74d9312b4db873983687"
Oct 30 22:46:53.491663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:53.491635    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h" event={"ID":"5d3c5bee-4af9-4346-b452-f1501a45b0b6","Type":"ContainerDied","Data":"4571cb1c36f2f4c3073f319973184d290e3f075ac9f8f991c37c1df91b4d61cb"}
Oct 30 22:46:53.491818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:53.491664    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"
Oct 30 22:46:53.491818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:53.491682    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4571cb1c36f2f4c3073f319973184d290e3f075ac9f8f991c37c1df91b4d61cb"
Oct 30 22:46:55.621606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.603610    4581 generic.go:356] "Generic (PLEG): container finished" podID="b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" containerID="76bce2d210285b28c7e1d8d81c95971983ae433c8fd24abb9e57178d0111592d" exitCode=0
Oct 30 22:46:55.621606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.604210    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956" event={"ID":"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc","Type":"ContainerDied","Data":"76bce2d210285b28c7e1d8d81c95971983ae433c8fd24abb9e57178d0111592d"}
Oct 30 22:46:55.621606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.616424    4581 generic.go:356] "Generic (PLEG): container finished" podID="df98ade7-2aa1-4bea-b2f2-6901e7564e5a" containerID="a4414354e6b1a3c46a0c873dd66faee563c7e7dfdc1dcfa317a1c868b6178ed0" exitCode=0
Oct 30 22:46:55.621606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.616614    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm" event={"ID":"df98ade7-2aa1-4bea-b2f2-6901e7564e5a","Type":"ContainerDied","Data":"a4414354e6b1a3c46a0c873dd66faee563c7e7dfdc1dcfa317a1c868b6178ed0"}
Oct 30 22:46:55.737017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.736982    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:46:55.822475 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.822436    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:46:55.833060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.833028    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-plugins-config-volume\") pod \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\" (UID: \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\") "
Oct 30 22:46:55.833266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.833109    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-kube-api-access-vh2d8\" (UniqueName: \"kubernetes.io/projected/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-kube-api-access-vh2d8\") pod \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\" (UID: \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\") "
Oct 30 22:46:55.833266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.833256    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-kube-api-access-jjz6c\" (UniqueName: \"kubernetes.io/projected/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-kube-api-access-jjz6c\") pod \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\" (UID: \"df98ade7-2aa1-4bea-b2f2-6901e7564e5a\") "
Oct 30 22:46:55.833504 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.833481    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-plugins-config-volume\") pod \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\" (UID: \"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc\") "
Oct 30 22:46:55.833883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.833854    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-plugins-config-volume" pod "df98ade7-2aa1-4bea-b2f2-6901e7564e5a" (UID: "df98ade7-2aa1-4bea-b2f2-6901e7564e5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:55.834020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.833985    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-plugins-config-volume" pod "b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" (UID: "b3c99d1f-cfcc-4c80-a316-aa96b27aeafc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:55.834187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.834170    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:55.834280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.834197    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:55.836125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.836057    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-kube-api-access-vh2d8" pod "b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" (UID: "b3c99d1f-cfcc-4c80-a316-aa96b27aeafc"). InnerVolumeSpecName "kube-api-access-vh2d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:55.836336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.836314    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-kube-api-access-jjz6c" pod "df98ade7-2aa1-4bea-b2f2-6901e7564e5a" (UID: "df98ade7-2aa1-4bea-b2f2-6901e7564e5a"). InnerVolumeSpecName "kube-api-access-jjz6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:55.935769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.935732    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2d8\" (UniqueName: \"kubernetes.io/projected/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc-kube-api-access-vh2d8\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:55.935769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:55.935766    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjz6c\" (UniqueName: \"kubernetes.io/projected/df98ade7-2aa1-4bea-b2f2-6901e7564e5a-kube-api-access-jjz6c\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:46:56.037606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.037567    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"]
Oct 30 22:46:56.042911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.042878    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cc7d52-4h7zt"]
Oct 30 22:46:56.101552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.101479    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d63e0e-b770-410a-a0f7-da8483a1803d" path="/var/lib/kubelet/pods/b7d63e0e-b770-410a-a0f7-da8483a1803d/volumes"
Oct 30 22:46:56.627798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.627769    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"
Oct 30 22:46:56.628326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.627796    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956" event={"ID":"b3c99d1f-cfcc-4c80-a316-aa96b27aeafc","Type":"ContainerDied","Data":"626f146bd41d994541d6e09c218b27c0341d5aaab75ad6d5590571862cd4d1ac"}
Oct 30 22:46:56.628326 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.627869    4581 scope.go:122] "RemoveContainer" containerID="76bce2d210285b28c7e1d8d81c95971983ae433c8fd24abb9e57178d0111592d"
Oct 30 22:46:56.629524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.629503    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"
Oct 30 22:46:56.629628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.629559    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm" event={"ID":"df98ade7-2aa1-4bea-b2f2-6901e7564e5a","Type":"ContainerDied","Data":"5099fb27cf9b100c3b5e260227031036c94bb52940cde906d7cb0bb74bd0e944"}
Oct 30 22:46:56.643067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.643047    4581 scope.go:122] "RemoveContainer" containerID="a4414354e6b1a3c46a0c873dd66faee563c7e7dfdc1dcfa317a1c868b6178ed0"
Oct 30 22:46:56.679207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.679168    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"]
Oct 30 22:46:56.684757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.684729    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-999470-99979b95c-2x956"]
Oct 30 22:46:56.712482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.712340    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"]
Oct 30 22:46:56.722172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:56.722131    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-999470-7757847d95-dfvnm"]
Oct 30 22:46:57.191703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:57.191674    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:46:57.196929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:57.196908    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:46:58.124243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:58.123814    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" path="/var/lib/kubelet/pods/b3c99d1f-cfcc-4c80-a316-aa96b27aeafc/volumes"
Oct 30 22:46:58.124787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:58.124422    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df98ade7-2aa1-4bea-b2f2-6901e7564e5a" path="/var/lib/kubelet/pods/df98ade7-2aa1-4bea-b2f2-6901e7564e5a/volumes"
Oct 30 22:46:59.689972 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:46:59.689940    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"]
Oct 30 22:47:00.306238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.305677    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"]
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.306822    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5d3c5bee-4af9-4346-b452-f1501a45b0b6" containerName="revoke-keys"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.306850    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3c5bee-4af9-4346-b452-f1501a45b0b6" containerName="revoke-keys"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.306895    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" containerName="payload-processing"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.306903    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" containerName="payload-processing"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.306932    4581 cpu_manager.go:414] "containerMap: removing container" podUID="df98ade7-2aa1-4bea-b2f2-6901e7564e5a" containerName="payload-pre-processing"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.306939    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df98ade7-2aa1-4bea-b2f2-6901e7564e5a" containerName="payload-pre-processing"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.306957    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29" containerName="revoke-keys"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.306964    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29" containerName="revoke-keys"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.307134    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3c5bee-4af9-4346-b452-f1501a45b0b6" containerName="revoke-keys"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.307151    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c99d1f-cfcc-4c80-a316-aa96b27aeafc" containerName="payload-processing"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.307164    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="df98ade7-2aa1-4bea-b2f2-6901e7564e5a" containerName="payload-pre-processing"
Oct 30 22:47:00.310307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.307176    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29" containerName="revoke-keys"
Oct 30 22:47:00.320729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.318114    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.342753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.342696    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"]
Oct 30 22:47:00.349332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.348919    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:47:00.349332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.348962    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:00.370941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.370905    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:00.371307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.371288    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:00.407899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.407406    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmp7q\" (UniqueName: \"kubernetes.io/projected/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-kube-api-access-qmp7q\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.407899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.407513    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.407899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.407560    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.508910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.508878    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmp7q\" (UniqueName: \"kubernetes.io/projected/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-kube-api-access-qmp7q\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.509130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.508941    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.509130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.508967    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.509631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.509604    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.511612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.511581    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.528613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.528587    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmp7q\" (UniqueName: \"kubernetes.io/projected/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-kube-api-access-qmp7q\") pod \"maas-api-revoke-sub-10f8d403-sd78r\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.732015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.731971    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:00.772634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.772596    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"]
Oct 30 22:47:00.779169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.779144    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:00.802208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.802163    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"]
Oct 30 22:47:00.922921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.922876    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmcdr\" (UniqueName: \"kubernetes.io/projected/c431ab82-888a-485e-8942-30a9718b4fd4-kube-api-access-pmcdr\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:00.923139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.923113    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:00.923312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:00.923294    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:01.024758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.024717    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:01.024956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.024851    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:01.025055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.025039    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcdr\" (UniqueName: \"kubernetes.io/projected/c431ab82-888a-485e-8942-30a9718b4fd4-kube-api-access-pmcdr\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:01.025482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.025458    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:01.027528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.027493    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:01.057274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.057249    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcdr\" (UniqueName: \"kubernetes.io/projected/c431ab82-888a-485e-8942-30a9718b4fd4-kube-api-access-pmcdr\") pod \"maas-api-revoke-sub-7737f85d-269fx\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:01.058226 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.058192    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"]
Oct 30 22:47:01.061027 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:47:01.060997    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6fddfb_fdae_46bf_a2c3_91ca389c2e25.slice/crio-62c1f435c942cf92ea9b4de23c50cf51a497443a79e9d451cbb11e9c70fee196 WatchSource:0}: Error finding container 62c1f435c942cf92ea9b4de23c50cf51a497443a79e9d451cbb11e9c70fee196: Status 404 returned error can't find the container with id 62c1f435c942cf92ea9b4de23c50cf51a497443a79e9d451cbb11e9c70fee196
Oct 30 22:47:01.098129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.098105    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:01.478478 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:47:01.478447    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc431ab82_888a_485e_8942_30a9718b4fd4.slice/crio-fc4c2a27276ba0b95ea42d613a67b9b98b1f51b3afc53c1041970669662a610c WatchSource:0}: Error finding container fc4c2a27276ba0b95ea42d613a67b9b98b1f51b3afc53c1041970669662a610c: Status 404 returned error can't find the container with id fc4c2a27276ba0b95ea42d613a67b9b98b1f51b3afc53c1041970669662a610c
Oct 30 22:47:01.479979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.479950    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"]
Oct 30 22:47:01.720480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.720441    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r" event={"ID":"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25","Type":"ContainerStarted","Data":"f9c82c244c3c24cc399dfea36d9e5d73549e22f4f5e0a3be7da5319656dd1963"}
Oct 30 22:47:01.720691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.720504    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r" event={"ID":"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25","Type":"ContainerStarted","Data":"62c1f435c942cf92ea9b4de23c50cf51a497443a79e9d451cbb11e9c70fee196"}
Oct 30 22:47:01.722279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.722251    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx" event={"ID":"c431ab82-888a-485e-8942-30a9718b4fd4","Type":"ContainerStarted","Data":"fc4c2a27276ba0b95ea42d613a67b9b98b1f51b3afc53c1041970669662a610c"}
Oct 30 22:47:01.766704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:01.766607    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r" podStartSLOduration=1.766593318 podStartE2EDuration="1.766593318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:01.753350421 +0000 UTC m=+2405.792289307" watchObservedRunningTime="2026-10-30 22:47:01.766593318 +0000 UTC m=+2405.805532205"
Oct 30 22:47:02.480963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:02.480931    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:47:02.481435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:02.481415    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:47:02.732839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:02.732742    4581 generic.go:356] "Generic (PLEG): container finished" podID="c431ab82-888a-485e-8942-30a9718b4fd4" containerID="38672501bde060c1d58ef89ea747889f6561b1a7387e8b45c80bdd6f2edb8a5b" exitCode=0
Oct 30 22:47:02.733025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:02.732831    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx" event={"ID":"c431ab82-888a-485e-8942-30a9718b4fd4","Type":"ContainerDied","Data":"38672501bde060c1d58ef89ea747889f6561b1a7387e8b45c80bdd6f2edb8a5b"}
Oct 30 22:47:02.734483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:02.734458    4581 generic.go:356] "Generic (PLEG): container finished" podID="2e6fddfb-fdae-46bf-a2c3-91ca389c2e25" containerID="f9c82c244c3c24cc399dfea36d9e5d73549e22f4f5e0a3be7da5319656dd1963" exitCode=0
Oct 30 22:47:02.734598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:02.734488    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r" event={"ID":"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25","Type":"ContainerDied","Data":"f9c82c244c3c24cc399dfea36d9e5d73549e22f4f5e0a3be7da5319656dd1963"}
Oct 30 22:47:03.995974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:03.995903    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:03.999858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:03.999836    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:04.145740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.145704    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-openshift-ca\") pod \"c431ab82-888a-485e-8942-30a9718b4fd4\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") "
Oct 30 22:47:04.145921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.145801    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-xks-ca\") pod \"c431ab82-888a-485e-8942-30a9718b4fd4\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") "
Oct 30 22:47:04.145921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.145832    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-openshift-ca\") pod \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") "
Oct 30 22:47:04.146117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.146082    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c431ab82-888a-485e-8942-30a9718b4fd4-kube-api-access-pmcdr\" (UniqueName: \"kubernetes.io/projected/c431ab82-888a-485e-8942-30a9718b4fd4-kube-api-access-pmcdr\") pod \"c431ab82-888a-485e-8942-30a9718b4fd4\" (UID: \"c431ab82-888a-485e-8942-30a9718b4fd4\") "
Oct 30 22:47:04.146186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.146141    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-xks-ca\") pod \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") "
Oct 30 22:47:04.146276 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.146192    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-openshift-ca" pod "c431ab82-888a-485e-8942-30a9718b4fd4" (UID: "c431ab82-888a-485e-8942-30a9718b4fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:04.146350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.146289    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-kube-api-access-qmp7q\" (UniqueName: \"kubernetes.io/projected/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-kube-api-access-qmp7q\") pod \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\" (UID: \"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25\") "
Oct 30 22:47:04.146350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.146291    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-openshift-ca" pod "2e6fddfb-fdae-46bf-a2c3-91ca389c2e25" (UID: "2e6fddfb-fdae-46bf-a2c3-91ca389c2e25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:04.146972 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.146948    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:04.146972 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.146971    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:04.148442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.148413    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-xks-ca" pod "c431ab82-888a-485e-8942-30a9718b4fd4" (UID: "c431ab82-888a-485e-8942-30a9718b4fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:04.148686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.148662    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-kube-api-access-qmp7q" pod "2e6fddfb-fdae-46bf-a2c3-91ca389c2e25" (UID: "2e6fddfb-fdae-46bf-a2c3-91ca389c2e25"). InnerVolumeSpecName "kube-api-access-qmp7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:04.148870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.148839    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-xks-ca" pod "2e6fddfb-fdae-46bf-a2c3-91ca389c2e25" (UID: "2e6fddfb-fdae-46bf-a2c3-91ca389c2e25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:04.148962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.148879    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c431ab82-888a-485e-8942-30a9718b4fd4-kube-api-access-pmcdr" pod "c431ab82-888a-485e-8942-30a9718b4fd4" (UID: "c431ab82-888a-485e-8942-30a9718b4fd4"). InnerVolumeSpecName "kube-api-access-pmcdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:04.249021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.248930    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmcdr\" (UniqueName: \"kubernetes.io/projected/c431ab82-888a-485e-8942-30a9718b4fd4-kube-api-access-pmcdr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:04.249021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.248967    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:04.249021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.248978    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmp7q\" (UniqueName: \"kubernetes.io/projected/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25-kube-api-access-qmp7q\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:04.249021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.248987    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c431ab82-888a-485e-8942-30a9718b4fd4-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:04.755060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.755030    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"
Oct 30 22:47:04.755321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.755068    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx" event={"ID":"c431ab82-888a-485e-8942-30a9718b4fd4","Type":"ContainerDied","Data":"fc4c2a27276ba0b95ea42d613a67b9b98b1f51b3afc53c1041970669662a610c"}
Oct 30 22:47:04.755321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.755127    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4c2a27276ba0b95ea42d613a67b9b98b1f51b3afc53c1041970669662a610c"
Oct 30 22:47:04.757028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.757004    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"
Oct 30 22:47:04.757186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.757036    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r" event={"ID":"2e6fddfb-fdae-46bf-a2c3-91ca389c2e25","Type":"ContainerDied","Data":"62c1f435c942cf92ea9b4de23c50cf51a497443a79e9d451cbb11e9c70fee196"}
Oct 30 22:47:04.757186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:04.757080    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62c1f435c942cf92ea9b4de23c50cf51a497443a79e9d451cbb11e9c70fee196"
Oct 30 22:47:08.512130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:08.512089    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"]
Oct 30 22:47:08.517082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:08.517054    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-999470-pxt22"]
Oct 30 22:47:10.101232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:10.101186    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46216be0-7d8d-4f70-95cf-af856b684f7b" path="/var/lib/kubelet/pods/46216be0-7d8d-4f70-95cf-af856b684f7b/volumes"
Oct 30 22:47:10.820483 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:47:10.820436    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:47:14.723357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:14.723313    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l" podUID="fcfc6689-f901-4959-aa31-1d11765ebc68" containerName="main" containerID="cri-o://086fc814c6f80a471411f66d18d9be7bc0ab589d3c822d746bc7140b880af06b" gracePeriod=45
Oct 30 22:47:14.859921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:14.859885    4581 generic.go:356] "Generic (PLEG): container finished" podID="fcfc6689-f901-4959-aa31-1d11765ebc68" containerID="086fc814c6f80a471411f66d18d9be7bc0ab589d3c822d746bc7140b880af06b" exitCode=0
Oct 30 22:47:14.860103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:14.859920    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l" event={"ID":"fcfc6689-f901-4959-aa31-1d11765ebc68","Type":"ContainerDied","Data":"086fc814c6f80a471411f66d18d9be7bc0ab589d3c822d746bc7140b880af06b"}
Oct 30 22:47:15.042549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.042522    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:47:15.173733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.173697    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-dshm\") pod \"fcfc6689-f901-4959-aa31-1d11765ebc68\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") "
Oct 30 22:47:15.173934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.173815    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-home\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-home\") pod \"fcfc6689-f901-4959-aa31-1d11765ebc68\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") "
Oct 30 22:47:15.173934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.173891    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcfc6689-f901-4959-aa31-1d11765ebc68-tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfc6689-f901-4959-aa31-1d11765ebc68-tls-certs\") pod \"fcfc6689-f901-4959-aa31-1d11765ebc68\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") "
Oct 30 22:47:15.173934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.173926    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-tmp-dir\") pod \"fcfc6689-f901-4959-aa31-1d11765ebc68\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") "
Oct 30 22:47:15.174124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.173962    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfc6689-f901-4959-aa31-1d11765ebc68-kube-api-access-kkgxs\" (UniqueName: \"kubernetes.io/projected/fcfc6689-f901-4959-aa31-1d11765ebc68-kube-api-access-kkgxs\") pod \"fcfc6689-f901-4959-aa31-1d11765ebc68\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") "
Oct 30 22:47:15.174124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.174001    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-model-cache\") pod \"fcfc6689-f901-4959-aa31-1d11765ebc68\" (UID: \"fcfc6689-f901-4959-aa31-1d11765ebc68\") "
Oct 30 22:47:15.174255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.174121    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-home" pod "fcfc6689-f901-4959-aa31-1d11765ebc68" (UID: "fcfc6689-f901-4959-aa31-1d11765ebc68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:15.174255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.174243    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-tmp-dir" pod "fcfc6689-f901-4959-aa31-1d11765ebc68" (UID: "fcfc6689-f901-4959-aa31-1d11765ebc68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:15.174679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.174649    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-model-cache" pod "fcfc6689-f901-4959-aa31-1d11765ebc68" (UID: "fcfc6689-f901-4959-aa31-1d11765ebc68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:15.175116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.174886    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.175303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.175279    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.178553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.177187    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfc6689-f901-4959-aa31-1d11765ebc68-tls-certs" pod "fcfc6689-f901-4959-aa31-1d11765ebc68" (UID: "fcfc6689-f901-4959-aa31-1d11765ebc68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:15.178553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.177199    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-dshm" pod "fcfc6689-f901-4959-aa31-1d11765ebc68" (UID: "fcfc6689-f901-4959-aa31-1d11765ebc68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:15.178553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.177304    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfc6689-f901-4959-aa31-1d11765ebc68-kube-api-access-kkgxs" pod "fcfc6689-f901-4959-aa31-1d11765ebc68" (UID: "fcfc6689-f901-4959-aa31-1d11765ebc68"). InnerVolumeSpecName "kube-api-access-kkgxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:15.277821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.277718    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.277821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.277758    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfc6689-f901-4959-aa31-1d11765ebc68-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.277821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.277774    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkgxs\" (UniqueName: \"kubernetes.io/projected/fcfc6689-f901-4959-aa31-1d11765ebc68-kube-api-access-kkgxs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.277821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.277787    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfc6689-f901-4959-aa31-1d11765ebc68-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:15.922245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.921085    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"
Oct 30 22:47:15.933250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.927338    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l" event={"ID":"fcfc6689-f901-4959-aa31-1d11765ebc68","Type":"ContainerDied","Data":"323cafba49730c47297263ebaf71df0da4cb20e30f2430e4fd32f11653d6bb37"}
Oct 30 22:47:15.933250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:15.927402    4581 scope.go:122] "RemoveContainer" containerID="086fc814c6f80a471411f66d18d9be7bc0ab589d3c822d746bc7140b880af06b"
Oct 30 22:47:16.008016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.007976    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"]
Oct 30 22:47:16.015274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.015242    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-40229e/test-model-40229e-kserve-66586c5b47-jfd7l"]
Oct 30 22:47:16.099134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.099098    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfc6689-f901-4959-aa31-1d11765ebc68" path="/var/lib/kubelet/pods/fcfc6689-f901-4959-aa31-1d11765ebc68/volumes"
Oct 30 22:47:16.160349 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.160319    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"]
Oct 30 22:47:16.161034 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161020    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2e6fddfb-fdae-46bf-a2c3-91ca389c2e25" containerName="revoke-keys"
Oct 30 22:47:16.161107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161037    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6fddfb-fdae-46bf-a2c3-91ca389c2e25" containerName="revoke-keys"
Oct 30 22:47:16.161107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161060    4581 cpu_manager.go:414] "containerMap: removing container" podUID="fcfc6689-f901-4959-aa31-1d11765ebc68" containerName="main"
Oct 30 22:47:16.161107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161065    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfc6689-f901-4959-aa31-1d11765ebc68" containerName="main"
Oct 30 22:47:16.161107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161093    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c431ab82-888a-485e-8942-30a9718b4fd4" containerName="revoke-keys"
Oct 30 22:47:16.161107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161099    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c431ab82-888a-485e-8942-30a9718b4fd4" containerName="revoke-keys"
Oct 30 22:47:16.161343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161208    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c431ab82-888a-485e-8942-30a9718b4fd4" containerName="revoke-keys"
Oct 30 22:47:16.161343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161242    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6fddfb-fdae-46bf-a2c3-91ca389c2e25" containerName="revoke-keys"
Oct 30 22:47:16.161343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.161255    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfc6689-f901-4959-aa31-1d11765ebc68" containerName="main"
Oct 30 22:47:16.166164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.166146    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.180391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.180370    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:16.180391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.180389    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:16.188743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.188712    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"]
Oct 30 22:47:16.198849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.198819    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:16.201372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.199379    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:47:16.206370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.206336    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.206749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.206721    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjzr\" (UniqueName: \"kubernetes.io/projected/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-kube-api-access-9bjzr\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.206882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.206861    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.308887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.308848    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.309122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.309009    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjzr\" (UniqueName: \"kubernetes.io/projected/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-kube-api-access-9bjzr\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.309122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.309041    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.309568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.309550    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.311947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.311916    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.326282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.326254    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjzr\" (UniqueName: \"kubernetes.io/projected/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-kube-api-access-9bjzr\") pod \"maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.482170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.481982    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:16.798116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.798086    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"]
Oct 30 22:47:16.799750 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:47:16.799721    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcf9887_057f_44d3_9e0f_1bc89c9cdaf6.slice/crio-33f6fe9c2d04249d030d52a262d72adbf10a52d517a3ac7a9eca98b5c945ab94 WatchSource:0}: Error finding container 33f6fe9c2d04249d030d52a262d72adbf10a52d517a3ac7a9eca98b5c945ab94: Status 404 returned error can't find the container with id 33f6fe9c2d04249d030d52a262d72adbf10a52d517a3ac7a9eca98b5c945ab94
Oct 30 22:47:16.939281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:16.939242    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg" event={"ID":"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6","Type":"ContainerStarted","Data":"33f6fe9c2d04249d030d52a262d72adbf10a52d517a3ac7a9eca98b5c945ab94"}
Oct 30 22:47:17.951566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:17.951526    4581 generic.go:356] "Generic (PLEG): container finished" podID="2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6" containerID="4cbbf0dcd417f2a3550d087337fa93379df193f3f3a855f41e2281b905838eed" exitCode=0
Oct 30 22:47:17.952028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:17.951611    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg" event={"ID":"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6","Type":"ContainerDied","Data":"4cbbf0dcd417f2a3550d087337fa93379df193f3f3a855f41e2281b905838eed"}
Oct 30 22:47:19.177714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.177688    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:19.258189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.258108    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-kube-api-access-9bjzr\" (UniqueName: \"kubernetes.io/projected/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-kube-api-access-9bjzr\") pod \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") "
Oct 30 22:47:19.258391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.258199    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-xks-ca\") pod \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") "
Oct 30 22:47:19.258476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.258422    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-openshift-ca\") pod \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\" (UID: \"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6\") "
Oct 30 22:47:19.258891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.258823    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-openshift-ca" pod "2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6" (UID: "2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:19.259375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.259347    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:19.260618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.260588    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-xks-ca" pod "2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6" (UID: "2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:19.260618 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.260588    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-kube-api-access-9bjzr" pod "2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6" (UID: "2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6"). InnerVolumeSpecName "kube-api-access-9bjzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:19.360904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.360870    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjzr\" (UniqueName: \"kubernetes.io/projected/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-kube-api-access-9bjzr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:19.360904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.360903    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:19.978094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.978056    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"
Oct 30 22:47:19.978355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.978212    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg" event={"ID":"2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6","Type":"ContainerDied","Data":"33f6fe9c2d04249d030d52a262d72adbf10a52d517a3ac7a9eca98b5c945ab94"}
Oct 30 22:47:19.978355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:19.978299    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33f6fe9c2d04249d030d52a262d72adbf10a52d517a3ac7a9eca98b5c945ab94"
Oct 30 22:47:20.276900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.276818    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"]
Oct 30 22:47:20.277757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.277734    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6" containerName="revoke-keys"
Oct 30 22:47:20.277864 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.277757    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6" containerName="revoke-keys"
Oct 30 22:47:20.277988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.277974    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6" containerName="revoke-keys"
Oct 30 22:47:20.281857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.281834    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.295378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.295349    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:20.295378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.295370    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:20.295556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.295413    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:47:20.295745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.295726    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:20.302370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.302346    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"]
Oct 30 22:47:20.403086 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.403044    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.403315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.403143    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.403315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.403280    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcztb\" (UniqueName: \"kubernetes.io/projected/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-kube-api-access-xcztb\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.505246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.505195    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcztb\" (UniqueName: \"kubernetes.io/projected/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-kube-api-access-xcztb\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.505522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.505503    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.505603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.505573    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.506140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.506122    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.508234 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.508195    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.522355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.522312    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcztb\" (UniqueName: \"kubernetes.io/projected/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-kube-api-access-xcztb\") pod \"maas-api-revoke-keys-e2e-inf-a-d85da4-n646j\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.596701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.596621    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:20.923588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:20.923559    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"]
Oct 30 22:47:20.924273 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:47:20.924243    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429d7d0b_0ab4_419a_9b79_56cda21bbdb2.slice/crio-819dd26a154b39d1214216fa72b8dd15f0db335073062fab4fc5ea0ead2857e7 WatchSource:0}: Error finding container 819dd26a154b39d1214216fa72b8dd15f0db335073062fab4fc5ea0ead2857e7: Status 404 returned error can't find the container with id 819dd26a154b39d1214216fa72b8dd15f0db335073062fab4fc5ea0ead2857e7
Oct 30 22:47:21.003431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:21.003392    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j" event={"ID":"429d7d0b-0ab4-419a-9b79-56cda21bbdb2","Type":"ContainerStarted","Data":"819dd26a154b39d1214216fa72b8dd15f0db335073062fab4fc5ea0ead2857e7"}
Oct 30 22:47:22.015392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:22.015358    4581 generic.go:356] "Generic (PLEG): container finished" podID="429d7d0b-0ab4-419a-9b79-56cda21bbdb2" containerID="4526a65401193e9f26fbbbc8c3c0814097ae5b8fc7b7cb7e61cc933fd3aa0bdb" exitCode=0
Oct 30 22:47:22.015858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:22.015445    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j" event={"ID":"429d7d0b-0ab4-419a-9b79-56cda21bbdb2","Type":"ContainerDied","Data":"4526a65401193e9f26fbbbc8c3c0814097ae5b8fc7b7cb7e61cc933fd3aa0bdb"}
Oct 30 22:47:23.291656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.291631    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:23.359382 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.359343    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-kube-api-access-xcztb\" (UniqueName: \"kubernetes.io/projected/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-kube-api-access-xcztb\") pod \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") "
Oct 30 22:47:23.359641 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.359504    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-xks-ca\") pod \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") "
Oct 30 22:47:23.359641 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.359576    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-openshift-ca\") pod \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\" (UID: \"429d7d0b-0ab4-419a-9b79-56cda21bbdb2\") "
Oct 30 22:47:23.361746 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.361717    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-openshift-ca" pod "429d7d0b-0ab4-419a-9b79-56cda21bbdb2" (UID: "429d7d0b-0ab4-419a-9b79-56cda21bbdb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:23.363186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.363146    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-kube-api-access-xcztb" pod "429d7d0b-0ab4-419a-9b79-56cda21bbdb2" (UID: "429d7d0b-0ab4-419a-9b79-56cda21bbdb2"). InnerVolumeSpecName "kube-api-access-xcztb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:23.365289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.365266    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-xks-ca" pod "429d7d0b-0ab4-419a-9b79-56cda21bbdb2" (UID: "429d7d0b-0ab4-419a-9b79-56cda21bbdb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:23.463141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.463106    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcztb\" (UniqueName: \"kubernetes.io/projected/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-kube-api-access-xcztb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:23.463141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.463145    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:23.463364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:23.463161    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429d7d0b-0ab4-419a-9b79-56cda21bbdb2-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:24.035091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.035058    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"
Oct 30 22:47:24.035306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.035095    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j" event={"ID":"429d7d0b-0ab4-419a-9b79-56cda21bbdb2","Type":"ContainerDied","Data":"819dd26a154b39d1214216fa72b8dd15f0db335073062fab4fc5ea0ead2857e7"}
Oct 30 22:47:24.035306 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.035161    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="819dd26a154b39d1214216fa72b8dd15f0db335073062fab4fc5ea0ead2857e7"
Oct 30 22:47:24.504471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.504432    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"]
Oct 30 22:47:24.505445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.505421    4581 cpu_manager.go:414] "containerMap: removing container" podUID="429d7d0b-0ab4-419a-9b79-56cda21bbdb2" containerName="revoke-keys"
Oct 30 22:47:24.505445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.505445    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429d7d0b-0ab4-419a-9b79-56cda21bbdb2" containerName="revoke-keys"
Oct 30 22:47:24.505665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.505650    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="429d7d0b-0ab4-419a-9b79-56cda21bbdb2" containerName="revoke-keys"
Oct 30 22:47:24.509887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.509859    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.522551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.522523    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:24.530420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.530394    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"]
Oct 30 22:47:24.531562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.531540    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:24.531562 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.531555    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:24.531765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.531584    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:47:24.582008 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.581967    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.582285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.582258    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.582455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.582303    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvzd\" (UniqueName: \"kubernetes.io/projected/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-kube-api-access-zwvzd\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.684365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.684325    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.684610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.684591    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.684703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.684631    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvzd\" (UniqueName: \"kubernetes.io/projected/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-kube-api-access-zwvzd\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.685609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.685578    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.687136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.687116    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.702075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.702050    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvzd\" (UniqueName: \"kubernetes.io/projected/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-kube-api-access-zwvzd\") pod \"maas-api-revoke-sub-5c457f51-rrmh6\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:24.829158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:24.829071    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:25.185435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:25.185407    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"]
Oct 30 22:47:25.187784 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:47:25.187754    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84a0652_844d_42d1_93b5_1c4cb5ac96cd.slice/crio-3c21d67b2471ad0ca36d9862c4400b839cb5f080486c3e792ea6577a8b60c275 WatchSource:0}: Error finding container 3c21d67b2471ad0ca36d9862c4400b839cb5f080486c3e792ea6577a8b60c275: Status 404 returned error can't find the container with id 3c21d67b2471ad0ca36d9862c4400b839cb5f080486c3e792ea6577a8b60c275
Oct 30 22:47:26.056839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:26.056796    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6" event={"ID":"c84a0652-844d-42d1-93b5-1c4cb5ac96cd","Type":"ContainerStarted","Data":"015b1a1ec5c1d7d6fcb0a36dc06c2f81f5536b9055a73fe419958f392e2bc468"}
Oct 30 22:47:26.057311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:26.056905    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6" event={"ID":"c84a0652-844d-42d1-93b5-1c4cb5ac96cd","Type":"ContainerStarted","Data":"3c21d67b2471ad0ca36d9862c4400b839cb5f080486c3e792ea6577a8b60c275"}
Oct 30 22:47:26.110662 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:26.110596    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6" podStartSLOduration=2.110578376 podStartE2EDuration="2.110578376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:26.103310135 +0000 UTC m=+2430.142249021" watchObservedRunningTime="2026-10-30 22:47:26.110578376 +0000 UTC m=+2430.149517261"
Oct 30 22:47:26.665730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:26.665698    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"]
Oct 30 22:47:26.699429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:26.699394    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"]
Oct 30 22:47:27.022792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.022698    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"]
Oct 30 22:47:27.075270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.073096    4581 generic.go:356] "Generic (PLEG): container finished" podID="c84a0652-844d-42d1-93b5-1c4cb5ac96cd" containerID="015b1a1ec5c1d7d6fcb0a36dc06c2f81f5536b9055a73fe419958f392e2bc468" exitCode=0
Oct 30 22:47:27.075270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.073169    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6" event={"ID":"c84a0652-844d-42d1-93b5-1c4cb5ac96cd","Type":"ContainerDied","Data":"015b1a1ec5c1d7d6fcb0a36dc06c2f81f5536b9055a73fe419958f392e2bc468"}
Oct 30 22:47:27.276514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.276241    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"]
Oct 30 22:47:27.313514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.313483    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.341810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.341770    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.354242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.346173    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"]
Oct 30 22:47:27.483245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.463951    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.483245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.464007    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4xdt\" (UniqueName: \"kubernetes.io/projected/a4afb333-c8b8-468c-953c-66326f449f6c-kube-api-access-n4xdt\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.483245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.464292    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.504628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.498735    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.565312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.565202    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.565312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.565274    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4xdt\" (UniqueName: \"kubernetes.io/projected/a4afb333-c8b8-468c-953c-66326f449f6c-kube-api-access-n4xdt\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.567883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.567847    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.584051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.583699    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4xdt\" (UniqueName: \"kubernetes.io/projected/a4afb333-c8b8-468c-953c-66326f449f6c-kube-api-access-n4xdt\") pod \"maas-api-revoke-keys-e2e-inf-a-40229e-tlm24\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:27.698373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:27.698334    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:28.017816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.017785    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"]
Oct 30 22:47:28.018339 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:47:28.018309    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4afb333_c8b8_468c_953c_66326f449f6c.slice/crio-f255c78d54bb5d520d9b3b25d4e619b0cd7afaff08e60a2417183c4bd531cc8b WatchSource:0}: Error finding container f255c78d54bb5d520d9b3b25d4e619b0cd7afaff08e60a2417183c4bd531cc8b: Status 404 returned error can't find the container with id f255c78d54bb5d520d9b3b25d4e619b0cd7afaff08e60a2417183c4bd531cc8b
Oct 30 22:47:28.100107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.100072    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24" event={"ID":"a4afb333-c8b8-468c-953c-66326f449f6c","Type":"ContainerStarted","Data":"f255c78d54bb5d520d9b3b25d4e619b0cd7afaff08e60a2417183c4bd531cc8b"}
Oct 30 22:47:28.255677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.255642    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:28.277136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.277108    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-xks-ca\") pod \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") "
Oct 30 22:47:28.277308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.277253    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-openshift-ca\") pod \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") "
Oct 30 22:47:28.277308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.277305    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-kube-api-access-zwvzd\" (UniqueName: \"kubernetes.io/projected/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-kube-api-access-zwvzd\") pod \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\" (UID: \"c84a0652-844d-42d1-93b5-1c4cb5ac96cd\") "
Oct 30 22:47:28.277885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.277805    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-openshift-ca" pod "c84a0652-844d-42d1-93b5-1c4cb5ac96cd" (UID: "c84a0652-844d-42d1-93b5-1c4cb5ac96cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:28.283247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.279761    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-xks-ca" pod "c84a0652-844d-42d1-93b5-1c4cb5ac96cd" (UID: "c84a0652-844d-42d1-93b5-1c4cb5ac96cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:28.283247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.282370    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-kube-api-access-zwvzd" pod "c84a0652-844d-42d1-93b5-1c4cb5ac96cd" (UID: "c84a0652-844d-42d1-93b5-1c4cb5ac96cd"). InnerVolumeSpecName "kube-api-access-zwvzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:28.379418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.379386    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:28.379418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.379419    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvzd\" (UniqueName: \"kubernetes.io/projected/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-kube-api-access-zwvzd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:28.379640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:28.379430    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a0652-844d-42d1-93b5-1c4cb5ac96cd-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:29.104172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:29.104139    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"
Oct 30 22:47:29.104603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:29.104172    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6" event={"ID":"c84a0652-844d-42d1-93b5-1c4cb5ac96cd","Type":"ContainerDied","Data":"3c21d67b2471ad0ca36d9862c4400b839cb5f080486c3e792ea6577a8b60c275"}
Oct 30 22:47:29.104603 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:29.104264    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c21d67b2471ad0ca36d9862c4400b839cb5f080486c3e792ea6577a8b60c275"
Oct 30 22:47:29.106036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:29.106012    4581 generic.go:356] "Generic (PLEG): container finished" podID="a4afb333-c8b8-468c-953c-66326f449f6c" containerID="e30412f2117afa533553d91ac1a05a80d06aa4353d2c8307518d4d68582e915d" exitCode=0
Oct 30 22:47:29.106184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:29.106132    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24" event={"ID":"a4afb333-c8b8-468c-953c-66326f449f6c","Type":"ContainerDied","Data":"e30412f2117afa533553d91ac1a05a80d06aa4353d2c8307518d4d68582e915d"}
Oct 30 22:47:30.290670 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.290644    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:30.310304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.310257    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4afb333-c8b8-468c-953c-66326f449f6c-kube-api-access-n4xdt\" (UniqueName: \"kubernetes.io/projected/a4afb333-c8b8-468c-953c-66326f449f6c-kube-api-access-n4xdt\") pod \"a4afb333-c8b8-468c-953c-66326f449f6c\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") "
Oct 30 22:47:30.310488 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.310390    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-xks-ca\") pod \"a4afb333-c8b8-468c-953c-66326f449f6c\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") "
Oct 30 22:47:30.310574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.310555    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-openshift-ca\") pod \"a4afb333-c8b8-468c-953c-66326f449f6c\" (UID: \"a4afb333-c8b8-468c-953c-66326f449f6c\") "
Oct 30 22:47:30.310983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.310949    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-openshift-ca" pod "a4afb333-c8b8-468c-953c-66326f449f6c" (UID: "a4afb333-c8b8-468c-953c-66326f449f6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:30.312570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.312541    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-xks-ca" pod "a4afb333-c8b8-468c-953c-66326f449f6c" (UID: "a4afb333-c8b8-468c-953c-66326f449f6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:30.312692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.312668    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4afb333-c8b8-468c-953c-66326f449f6c-kube-api-access-n4xdt" pod "a4afb333-c8b8-468c-953c-66326f449f6c" (UID: "a4afb333-c8b8-468c-953c-66326f449f6c"). InnerVolumeSpecName "kube-api-access-n4xdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:30.412398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.412316    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:30.412398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.412345    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4xdt\" (UniqueName: \"kubernetes.io/projected/a4afb333-c8b8-468c-953c-66326f449f6c-kube-api-access-n4xdt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:30.412398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:30.412355    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4afb333-c8b8-468c-953c-66326f449f6c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:31.160289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:31.160212    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"
Oct 30 22:47:31.160484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:31.160249    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24" event={"ID":"a4afb333-c8b8-468c-953c-66326f449f6c","Type":"ContainerDied","Data":"f255c78d54bb5d520d9b3b25d4e619b0cd7afaff08e60a2417183c4bd531cc8b"}
Oct 30 22:47:31.160484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:31.160393    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f255c78d54bb5d520d9b3b25d4e619b0cd7afaff08e60a2417183c4bd531cc8b"
Oct 30 22:47:31.671244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:31.670741    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn" podUID="710874c3-6407-406c-9955-5c3b9c2aa2c0" containerName="payload-processing" containerID="cri-o://87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9" gracePeriod=25
Oct 30 22:47:31.700002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:31.699960    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m" podUID="5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" containerName="payload-pre-processing" containerID="cri-o://762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5" gracePeriod=25
Oct 30 22:47:31.981447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:31.981415    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:47:32.060678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.060643    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710874c3-6407-406c-9955-5c3b9c2aa2c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710874c3-6407-406c-9955-5c3b9c2aa2c0-plugins-config-volume\") pod \"710874c3-6407-406c-9955-5c3b9c2aa2c0\" (UID: \"710874c3-6407-406c-9955-5c3b9c2aa2c0\") "
Oct 30 22:47:32.060880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.060714    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710874c3-6407-406c-9955-5c3b9c2aa2c0-kube-api-access-psmsr\" (UniqueName: \"kubernetes.io/projected/710874c3-6407-406c-9955-5c3b9c2aa2c0-kube-api-access-psmsr\") pod \"710874c3-6407-406c-9955-5c3b9c2aa2c0\" (UID: \"710874c3-6407-406c-9955-5c3b9c2aa2c0\") "
Oct 30 22:47:32.061134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.061101    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710874c3-6407-406c-9955-5c3b9c2aa2c0-plugins-config-volume" pod "710874c3-6407-406c-9955-5c3b9c2aa2c0" (UID: "710874c3-6407-406c-9955-5c3b9c2aa2c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:32.061586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.061557    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710874c3-6407-406c-9955-5c3b9c2aa2c0-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:32.063771 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.063739    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710874c3-6407-406c-9955-5c3b9c2aa2c0-kube-api-access-psmsr" pod "710874c3-6407-406c-9955-5c3b9c2aa2c0" (UID: "710874c3-6407-406c-9955-5c3b9c2aa2c0"). InnerVolumeSpecName "kube-api-access-psmsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:32.164023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.163993    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psmsr\" (UniqueName: \"kubernetes.io/projected/710874c3-6407-406c-9955-5c3b9c2aa2c0-kube-api-access-psmsr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:32.176235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.176185    4581 generic.go:356] "Generic (PLEG): container finished" podID="710874c3-6407-406c-9955-5c3b9c2aa2c0" containerID="87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9" exitCode=0
Oct 30 22:47:32.176413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.176260    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn" event={"ID":"710874c3-6407-406c-9955-5c3b9c2aa2c0","Type":"ContainerDied","Data":"87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9"}
Oct 30 22:47:32.176413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.176297    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"
Oct 30 22:47:32.176413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.176313    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn" event={"ID":"710874c3-6407-406c-9955-5c3b9c2aa2c0","Type":"ContainerDied","Data":"79bed5c52cdecec822c759aea00cfcc08456b1f45c2c2ada080653878ac3007c"}
Oct 30 22:47:32.176413 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.176344    4581 scope.go:122] "RemoveContainer" containerID="87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9"
Oct 30 22:47:32.191803 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.191782    4581 scope.go:122] "RemoveContainer" containerID="87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9"
Oct 30 22:47:32.192162 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:47:32.192139    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9\": container with ID starting with 87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9 not found: ID does not exist" containerID="87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9"
Oct 30 22:47:32.192271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.192172    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9"} err="failed to get container status \"87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9\": rpc error: code = NotFound desc = could not find container \"87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9\": container with ID starting with 87d315bb18754e746202912ad1e60c7f9f3280d79ed54b3e8d53ba0681186ba9 not found: ID does not exist"
Oct 30 22:47:32.223577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.223506    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"]
Oct 30 22:47:32.228532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:32.228505    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-54c7f0-5448fb4d5c-nrrpn"]
Oct 30 22:47:34.098137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:34.098102    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710874c3-6407-406c-9955-5c3b9c2aa2c0" path="/var/lib/kubelet/pods/710874c3-6407-406c-9955-5c3b9c2aa2c0/volumes"
Oct 30 22:47:37.109361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:37.109318    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m" podUID="5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:38.468787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:38.468751    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"]
Oct 30 22:47:40.203560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:40.203522    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"]
Oct 30 22:47:40.224527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:40.224491    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d85da4-n646j"]
Oct 30 22:47:40.817760 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:47:40.817723    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:47:42.055250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.055139    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr" podUID="b1322a3c-9054-4605-987a-1fd1fb53f46b" containerName="main" containerID="cri-o://a21ed1dfce311ccb68dcb74a7171d6e0f9269721c30b6704a586c1b71f780b87" gracePeriod=45
Oct 30 22:47:42.104446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.104414    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429d7d0b-0ab4-419a-9b79-56cda21bbdb2" path="/var/lib/kubelet/pods/429d7d0b-0ab4-419a-9b79-56cda21bbdb2/volumes"
Oct 30 22:47:42.347712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.347186    4581 generic.go:356] "Generic (PLEG): container finished" podID="b1322a3c-9054-4605-987a-1fd1fb53f46b" containerID="a21ed1dfce311ccb68dcb74a7171d6e0f9269721c30b6704a586c1b71f780b87" exitCode=0
Oct 30 22:47:42.347712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.347594    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr" event={"ID":"b1322a3c-9054-4605-987a-1fd1fb53f46b","Type":"ContainerDied","Data":"a21ed1dfce311ccb68dcb74a7171d6e0f9269721c30b6704a586c1b71f780b87"}
Oct 30 22:47:42.417839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.417803    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:47:42.481301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.481270    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-tmp-dir\") pod \"b1322a3c-9054-4605-987a-1fd1fb53f46b\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") "
Oct 30 22:47:42.481608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.481576    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-tmp-dir" pod "b1322a3c-9054-4605-987a-1fd1fb53f46b" (UID: "b1322a3c-9054-4605-987a-1fd1fb53f46b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:42.481726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.481705    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1322a3c-9054-4605-987a-1fd1fb53f46b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1322a3c-9054-4605-987a-1fd1fb53f46b-tls-certs\") pod \"b1322a3c-9054-4605-987a-1fd1fb53f46b\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") "
Oct 30 22:47:42.481865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.481849    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-home\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-home\") pod \"b1322a3c-9054-4605-987a-1fd1fb53f46b\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") "
Oct 30 22:47:42.482062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.482044    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1322a3c-9054-4605-987a-1fd1fb53f46b-kube-api-access-smhzx\" (UniqueName: \"kubernetes.io/projected/b1322a3c-9054-4605-987a-1fd1fb53f46b-kube-api-access-smhzx\") pod \"b1322a3c-9054-4605-987a-1fd1fb53f46b\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") "
Oct 30 22:47:42.482350 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.482327    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-model-cache\") pod \"b1322a3c-9054-4605-987a-1fd1fb53f46b\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") "
Oct 30 22:47:42.482650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.482128    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-home" pod "b1322a3c-9054-4605-987a-1fd1fb53f46b" (UID: "b1322a3c-9054-4605-987a-1fd1fb53f46b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:42.482650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.482550    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-model-cache" pod "b1322a3c-9054-4605-987a-1fd1fb53f46b" (UID: "b1322a3c-9054-4605-987a-1fd1fb53f46b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:42.482866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.482847    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-dshm\") pod \"b1322a3c-9054-4605-987a-1fd1fb53f46b\" (UID: \"b1322a3c-9054-4605-987a-1fd1fb53f46b\") "
Oct 30 22:47:42.483980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.483961    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:42.484115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.484102    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:42.484370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.484356    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:42.484658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.484628    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1322a3c-9054-4605-987a-1fd1fb53f46b-tls-certs" pod "b1322a3c-9054-4605-987a-1fd1fb53f46b" (UID: "b1322a3c-9054-4605-987a-1fd1fb53f46b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:42.484930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.484904    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1322a3c-9054-4605-987a-1fd1fb53f46b-kube-api-access-smhzx" pod "b1322a3c-9054-4605-987a-1fd1fb53f46b" (UID: "b1322a3c-9054-4605-987a-1fd1fb53f46b"). InnerVolumeSpecName "kube-api-access-smhzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:42.486743 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.486722    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-dshm" pod "b1322a3c-9054-4605-987a-1fd1fb53f46b" (UID: "b1322a3c-9054-4605-987a-1fd1fb53f46b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:42.585700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.585666    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1322a3c-9054-4605-987a-1fd1fb53f46b-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:42.585700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.585699    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1322a3c-9054-4605-987a-1fd1fb53f46b-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:42.585992 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:42.585714    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smhzx\" (UniqueName: \"kubernetes.io/projected/b1322a3c-9054-4605-987a-1fd1fb53f46b-kube-api-access-smhzx\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:43.362483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:43.362446    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"
Oct 30 22:47:43.362483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:43.362475    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr" event={"ID":"b1322a3c-9054-4605-987a-1fd1fb53f46b","Type":"ContainerDied","Data":"fd27dd7fc822390c659458a65ae4a11d910732f0778be9f6c7b95f2b3def06af"}
Oct 30 22:47:43.363027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:43.362535    4581 scope.go:122] "RemoveContainer" containerID="a21ed1dfce311ccb68dcb74a7171d6e0f9269721c30b6704a586c1b71f780b87"
Oct 30 22:47:43.409289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:43.409258    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"]
Oct 30 22:47:43.418188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:43.418163    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c13db0/test-model-c13db0-kserve-6f84486fd9-hxkbr"]
Oct 30 22:47:43.469237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:43.469177    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps" podUID="ea375e9e-122e-486c-9da0-9e731d72db29" containerName="payload-pre-processing" containerID="cri-o://55c9d9ad647f2d9b27e1c900e6c20021d768357fb0fe852d6fb7803ceae0cd2c" gracePeriod=25
Oct 30 22:47:44.109059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:44.109021    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1322a3c-9054-4605-987a-1fd1fb53f46b" path="/var/lib/kubelet/pods/b1322a3c-9054-4605-987a-1fd1fb53f46b/volumes"
Oct 30 22:47:47.094354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:47.094318    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m" podUID="5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:47.656639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:47.656604    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"]
Oct 30 22:47:47.657173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:47.657144    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy" containerID="cri-o://b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59" gracePeriod=30
Oct 30 22:47:48.266369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.266333    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"]
Oct 30 22:47:48.267448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267426    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a4afb333-c8b8-468c-953c-66326f449f6c" containerName="revoke-keys"
Oct 30 22:47:48.267517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267450    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4afb333-c8b8-468c-953c-66326f449f6c" containerName="revoke-keys"
Oct 30 22:47:48.267517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267473    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c84a0652-844d-42d1-93b5-1c4cb5ac96cd" containerName="revoke-keys"
Oct 30 22:47:48.267517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267480    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84a0652-844d-42d1-93b5-1c4cb5ac96cd" containerName="revoke-keys"
Oct 30 22:47:48.267517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267508    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b1322a3c-9054-4605-987a-1fd1fb53f46b" containerName="main"
Oct 30 22:47:48.267517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267516    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1322a3c-9054-4605-987a-1fd1fb53f46b" containerName="main"
Oct 30 22:47:48.267683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267581    4581 cpu_manager.go:414] "containerMap: removing container" podUID="710874c3-6407-406c-9955-5c3b9c2aa2c0" containerName="payload-processing"
Oct 30 22:47:48.267683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267589    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710874c3-6407-406c-9955-5c3b9c2aa2c0" containerName="payload-processing"
Oct 30 22:47:48.267754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267747    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="710874c3-6407-406c-9955-5c3b9c2aa2c0" containerName="payload-processing"
Oct 30 22:47:48.267786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267763    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4afb333-c8b8-468c-953c-66326f449f6c" containerName="revoke-keys"
Oct 30 22:47:48.267786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267775    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84a0652-844d-42d1-93b5-1c4cb5ac96cd" containerName="revoke-keys"
Oct 30 22:47:48.267843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.267787    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1322a3c-9054-4605-987a-1fd1fb53f46b" containerName="main"
Oct 30 22:47:48.273471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.273453    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.299506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.299482    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"]
Oct 30 22:47:48.300808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.300781    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:48.300918 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.300828    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:48.307165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.307140    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:48.307346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.307208    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:47:48.401071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.401036    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.401260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.401100    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.401348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.401333    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtm26\" (UniqueName: \"kubernetes.io/projected/0f95efa3-f23a-4499-be2a-32ef49ae2205-kube-api-access-dtm26\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.503749 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.503702    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtm26\" (UniqueName: \"kubernetes.io/projected/0f95efa3-f23a-4499-be2a-32ef49ae2205-kube-api-access-dtm26\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.503927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.503900    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.504012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.503963    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.507551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.507517    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.508589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.508558    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.525188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.525155    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtm26\" (UniqueName: \"kubernetes.io/projected/0f95efa3-f23a-4499-be2a-32ef49ae2205-kube-api-access-dtm26\") pod \"maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.588912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.588874    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:48.907302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:48.907269    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"]
Oct 30 22:47:48.910086 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:47:48.909032    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f95efa3_f23a_4499_be2a_32ef49ae2205.slice/crio-ffbaf7f21ed42f90297b8283843a1898cb4e785976787769208d13e51294b551 WatchSource:0}: Error finding container ffbaf7f21ed42f90297b8283843a1898cb4e785976787769208d13e51294b551: Status 404 returned error can't find the container with id ffbaf7f21ed42f90297b8283843a1898cb4e785976787769208d13e51294b551
Oct 30 22:47:49.030627 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:49.030591    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"]
Oct 30 22:47:49.035238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:49.035206    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2884a3-jfq4b"]
Oct 30 22:47:49.450340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:49.449405    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq" event={"ID":"0f95efa3-f23a-4499-be2a-32ef49ae2205","Type":"ContainerStarted","Data":"baf73e6aff93e58d7e696b7287cf0c3a09a6e6087ad21b8505b6abc8ac435cfc"}
Oct 30 22:47:49.450340 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:49.449532    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq" event={"ID":"0f95efa3-f23a-4499-be2a-32ef49ae2205","Type":"ContainerStarted","Data":"ffbaf7f21ed42f90297b8283843a1898cb4e785976787769208d13e51294b551"}
Oct 30 22:47:49.489305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:49.489239    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq" podStartSLOduration=1.489200675 podStartE2EDuration="1.489200675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:49.485583111 +0000 UTC m=+2453.524522000" watchObservedRunningTime="2026-10-30 22:47:49.489200675 +0000 UTC m=+2453.528139562"
Oct 30 22:47:50.100092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:50.100058    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023343d0-b093-49e2-9a63-d6e9922940cc" path="/var/lib/kubelet/pods/023343d0-b093-49e2-9a63-d6e9922940cc/volumes"
Oct 30 22:47:50.461804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:50.461746    4581 generic.go:356] "Generic (PLEG): container finished" podID="0f95efa3-f23a-4499-be2a-32ef49ae2205" containerID="baf73e6aff93e58d7e696b7287cf0c3a09a6e6087ad21b8505b6abc8ac435cfc" exitCode=0
Oct 30 22:47:50.461804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:50.461804    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq" event={"ID":"0f95efa3-f23a-4499-be2a-32ef49ae2205","Type":"ContainerDied","Data":"baf73e6aff93e58d7e696b7287cf0c3a09a6e6087ad21b8505b6abc8ac435cfc"}
Oct 30 22:47:51.208230 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.208170    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps" podUID="ea375e9e-122e-486c-9da0-9e731d72db29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:51.324416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.324365    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"]
Oct 30 22:47:51.368243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.352313    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.376497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.376461    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.376895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.376866    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5kt\" (UniqueName: \"kubernetes.io/projected/78e2cbbd-e9fc-43b5-a683-b3901acb948e-kube-api-access-kz5kt\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.377110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.377095    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.379502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.379475    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"]
Oct 30 22:47:51.478631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.478596    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5kt\" (UniqueName: \"kubernetes.io/projected/78e2cbbd-e9fc-43b5-a683-b3901acb948e-kube-api-access-kz5kt\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.479360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.478723    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.479360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.478908    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.479965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.479935    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.481667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.481637    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.496140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.496108    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5kt\" (UniqueName: \"kubernetes.io/projected/78e2cbbd-e9fc-43b5-a683-b3901acb948e-kube-api-access-kz5kt\") pod \"maas-api-revoke-sub-3ae2484d-cvvhd\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.663517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.663481    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:51.693147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.693117    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:51.784891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.784859    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-openshift-ca\") pod \"0f95efa3-f23a-4499-be2a-32ef49ae2205\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") "
Oct 30 22:47:51.785007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.784927    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f95efa3-f23a-4499-be2a-32ef49ae2205-kube-api-access-dtm26\" (UniqueName: \"kubernetes.io/projected/0f95efa3-f23a-4499-be2a-32ef49ae2205-kube-api-access-dtm26\") pod \"0f95efa3-f23a-4499-be2a-32ef49ae2205\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") "
Oct 30 22:47:51.785264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.785242    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-xks-ca\") pod \"0f95efa3-f23a-4499-be2a-32ef49ae2205\" (UID: \"0f95efa3-f23a-4499-be2a-32ef49ae2205\") "
Oct 30 22:47:51.786587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.786546    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-openshift-ca" pod "0f95efa3-f23a-4499-be2a-32ef49ae2205" (UID: "0f95efa3-f23a-4499-be2a-32ef49ae2205"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:51.788495 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.788469    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-xks-ca" pod "0f95efa3-f23a-4499-be2a-32ef49ae2205" (UID: "0f95efa3-f23a-4499-be2a-32ef49ae2205"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:51.789199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.789170    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f95efa3-f23a-4499-be2a-32ef49ae2205-kube-api-access-dtm26" pod "0f95efa3-f23a-4499-be2a-32ef49ae2205" (UID: "0f95efa3-f23a-4499-be2a-32ef49ae2205"). InnerVolumeSpecName "kube-api-access-dtm26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:51.887542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.887502    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:51.887542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.887544    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f95efa3-f23a-4499-be2a-32ef49ae2205-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:51.887744 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.887559    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtm26\" (UniqueName: \"kubernetes.io/projected/0f95efa3-f23a-4499-be2a-32ef49ae2205-kube-api-access-dtm26\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:51.987070 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:51.987035    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:47:52.033626 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:47:52.033595    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e2cbbd_e9fc_43b5_a683_b3901acb948e.slice/crio-b4462fdbdeaf198b76a579d26a4ff2faaa3c29ed24b88a57a61f3f58901370e8 WatchSource:0}: Error finding container b4462fdbdeaf198b76a579d26a4ff2faaa3c29ed24b88a57a61f3f58901370e8: Status 404 returned error can't find the container with id b4462fdbdeaf198b76a579d26a4ff2faaa3c29ed24b88a57a61f3f58901370e8
Oct 30 22:47:52.035831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.035804    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"]
Oct 30 22:47:52.093352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.091752    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-plugins-config-volume\") pod \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\" (UID: \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\") "
Oct 30 22:47:52.093352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.091875    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-kube-api-access-qrrtq\" (UniqueName: \"kubernetes.io/projected/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-kube-api-access-qrrtq\") pod \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\" (UID: \"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66\") "
Oct 30 22:47:52.093892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.093860    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-plugins-config-volume" pod "5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" (UID: "5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:52.096257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.096181    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-kube-api-access-qrrtq" pod "5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" (UID: "5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66"). InnerVolumeSpecName "kube-api-access-qrrtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:52.194244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.193988    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:52.194244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.194037    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrtq\" (UniqueName: \"kubernetes.io/projected/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66-kube-api-access-qrrtq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:52.482953 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.482911    4581 generic.go:356] "Generic (PLEG): container finished" podID="5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" containerID="762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5" exitCode=0
Oct 30 22:47:52.483490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.482975    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"
Oct 30 22:47:52.483490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.482992    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m" event={"ID":"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66","Type":"ContainerDied","Data":"762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5"}
Oct 30 22:47:52.483490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.483052    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m" event={"ID":"5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66","Type":"ContainerDied","Data":"ee2cf6fc94da8fcde64504ea50e868a6c4b4904802e1a7e5a5d3fc57dfe9a7a7"}
Oct 30 22:47:52.483490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.483082    4581 scope.go:122] "RemoveContainer" containerID="762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5"
Oct 30 22:47:52.485687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.485423    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd" event={"ID":"78e2cbbd-e9fc-43b5-a683-b3901acb948e","Type":"ContainerStarted","Data":"c0d7afa65581be7dabe579bbe3147616aa8c8dd04112dc85d0da2592e3565b4a"}
Oct 30 22:47:52.485687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.485477    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd" event={"ID":"78e2cbbd-e9fc-43b5-a683-b3901acb948e","Type":"ContainerStarted","Data":"b4462fdbdeaf198b76a579d26a4ff2faaa3c29ed24b88a57a61f3f58901370e8"}
Oct 30 22:47:52.487439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.487415    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"
Oct 30 22:47:52.487620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.487586    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq" event={"ID":"0f95efa3-f23a-4499-be2a-32ef49ae2205","Type":"ContainerDied","Data":"ffbaf7f21ed42f90297b8283843a1898cb4e785976787769208d13e51294b551"}
Oct 30 22:47:52.487760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.487661    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffbaf7f21ed42f90297b8283843a1898cb4e785976787769208d13e51294b551"
Oct 30 22:47:52.511990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.511966    4581 scope.go:122] "RemoveContainer" containerID="762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5"
Oct 30 22:47:52.512327 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:47:52.512307    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5\": container with ID starting with 762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5 not found: ID does not exist" containerID="762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5"
Oct 30 22:47:52.512398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.512336    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5"} err="failed to get container status \"762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5\": rpc error: code = NotFound desc = could not find container \"762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5\": container with ID starting with 762ef2ef07301e7bbbbe58354a8f975b822fcb492fa08e0aaa43fe570145c8d5 not found: ID does not exist"
Oct 30 22:47:52.535585 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.535494    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd" podStartSLOduration=1.535480149 podStartE2EDuration="1.535480149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:52.532509393 +0000 UTC m=+2456.571448280" watchObservedRunningTime="2026-10-30 22:47:52.535480149 +0000 UTC m=+2456.574419084"
Oct 30 22:47:52.578993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.578961    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"]
Oct 30 22:47:52.584537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.584458    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-54c7f0-545cb4dc95-xpm4m"]
Oct 30 22:47:52.967913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:52.967885    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.022613    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-certs\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.022756    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-credential-socket\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.022801    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-envoy\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.022858    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-socket\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.022983    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-token\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-token\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.022979    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-certs" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.023149    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-podinfo\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.023200    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istiod-ca-cert\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.023253    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-data\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.023324    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-kube-api-access-9mbfh\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-kube-api-access-9mbfh\") pod \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\" (UID: \"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56\") "
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.023844    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-data" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.023871    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-credential-socket" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.024242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.024101    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-socket" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.026902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.024262    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.026902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.024284    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.026902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.024297    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.026902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.024311    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.026902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.024373    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istiod-ca-cert" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:53.039664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.037645    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-token" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:53.039664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.037782    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-envoy" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:53.041247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.040094    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-podinfo" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:53.064020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.063977    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-kube-api-access-9mbfh" pod "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" (UID: "6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56"). InnerVolumeSpecName "kube-api-access-9mbfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:53.125837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.125805    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.125837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.125836    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.125837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.125854    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.126136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.125868    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbfh\" (UniqueName: \"kubernetes.io/projected/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-kube-api-access-9mbfh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.126136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.125880    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:53.505724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.505684    4581 generic.go:356] "Generic (PLEG): container finished" podID="78e2cbbd-e9fc-43b5-a683-b3901acb948e" containerID="c0d7afa65581be7dabe579bbe3147616aa8c8dd04112dc85d0da2592e3565b4a" exitCode=0
Oct 30 22:47:53.506257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.505816    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd" event={"ID":"78e2cbbd-e9fc-43b5-a683-b3901acb948e","Type":"ContainerDied","Data":"c0d7afa65581be7dabe579bbe3147616aa8c8dd04112dc85d0da2592e3565b4a"}
Oct 30 22:47:53.508279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.508209    4581 generic.go:356] "Generic (PLEG): container finished" podID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerID="b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59" exitCode=0
Oct 30 22:47:53.508279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.508270    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"
Oct 30 22:47:53.508446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.508263    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" event={"ID":"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56","Type":"ContainerDied","Data":"b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59"}
Oct 30 22:47:53.508446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.508401    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz" event={"ID":"6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56","Type":"ContainerDied","Data":"0596d3f400dac42b622462a98ba450bc5c4714152b6da77fab9a1acfcf143cb9"}
Oct 30 22:47:53.508446 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.508436    4581 scope.go:122] "RemoveContainer" containerID="b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59"
Oct 30 22:47:53.522950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.522921    4581 scope.go:122] "RemoveContainer" containerID="fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0"
Oct 30 22:47:53.535529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.535508    4581 scope.go:122] "RemoveContainer" containerID="b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59"
Oct 30 22:47:53.535816 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:47:53.535796    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59\": container with ID starting with b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59 not found: ID does not exist" containerID="b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59"
Oct 30 22:47:53.535866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.535829    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59"} err="failed to get container status \"b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59\": rpc error: code = NotFound desc = could not find container \"b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59\": container with ID starting with b5a6940b6455fab1705715d7ff1e4024c1106cd7d8fa4e3c0acd7fb4e448eb59 not found: ID does not exist"
Oct 30 22:47:53.535866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.535851    4581 scope.go:122] "RemoveContainer" containerID="fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0"
Oct 30 22:47:53.536115 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:47:53.536093    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0\": container with ID starting with fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0 not found: ID does not exist" containerID="fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0"
Oct 30 22:47:53.536167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.536123    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0"} err="failed to get container status \"fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0\": rpc error: code = NotFound desc = could not find container \"fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0\": container with ID starting with fea6e9dedf9671eaaf18c44fd6ce9b8b40160bfff95fb7b1419dc9c7e0c072a0 not found: ID does not exist"
Oct 30 22:47:53.579389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.579356    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"]
Oct 30 22:47:53.589294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:53.589262    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d85da4-openshift-default-6b5f89667-d8fjz"]
Oct 30 22:47:54.103427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:54.103394    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" path="/var/lib/kubelet/pods/5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66/volumes"
Oct 30 22:47:54.104042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:54.104021    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" path="/var/lib/kubelet/pods/6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56/volumes"
Oct 30 22:47:54.837855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:54.837412    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:54.996940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:54.996890    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-xks-ca\") pod \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") "
Oct 30 22:47:54.997146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:54.997033    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-openshift-ca\") pod \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") "
Oct 30 22:47:54.997146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:54.997082    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e2cbbd-e9fc-43b5-a683-b3901acb948e-kube-api-access-kz5kt\" (UniqueName: \"kubernetes.io/projected/78e2cbbd-e9fc-43b5-a683-b3901acb948e-kube-api-access-kz5kt\") pod \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\" (UID: \"78e2cbbd-e9fc-43b5-a683-b3901acb948e\") "
Oct 30 22:47:54.997811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:54.997778    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-openshift-ca" pod "78e2cbbd-e9fc-43b5-a683-b3901acb948e" (UID: "78e2cbbd-e9fc-43b5-a683-b3901acb948e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:55.001391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:55.001348    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e2cbbd-e9fc-43b5-a683-b3901acb948e-kube-api-access-kz5kt" pod "78e2cbbd-e9fc-43b5-a683-b3901acb948e" (UID: "78e2cbbd-e9fc-43b5-a683-b3901acb948e"). InnerVolumeSpecName "kube-api-access-kz5kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:55.005392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:55.005324    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-xks-ca" pod "78e2cbbd-e9fc-43b5-a683-b3901acb948e" (UID: "78e2cbbd-e9fc-43b5-a683-b3901acb948e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:55.109076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:55.106710    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:55.109076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:55.106755    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz5kt\" (UniqueName: \"kubernetes.io/projected/78e2cbbd-e9fc-43b5-a683-b3901acb948e-kube-api-access-kz5kt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:55.109076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:55.106770    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e2cbbd-e9fc-43b5-a683-b3901acb948e-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:47:55.542920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:55.542842    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"
Oct 30 22:47:55.542920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:55.542863    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd" event={"ID":"78e2cbbd-e9fc-43b5-a683-b3901acb948e","Type":"ContainerDied","Data":"b4462fdbdeaf198b76a579d26a4ff2faaa3c29ed24b88a57a61f3f58901370e8"}
Oct 30 22:47:55.543123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:47:55.542919    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4462fdbdeaf198b76a579d26a4ff2faaa3c29ed24b88a57a61f3f58901370e8"
Oct 30 22:48:00.044151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:00.044117    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"]
Oct 30 22:48:00.053425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:00.053397    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55e2f1b-2tmbn"]
Oct 30 22:48:00.100173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:00.100135    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fff5de9-809a-446c-aa19-79020a584b4a" path="/var/lib/kubelet/pods/7fff5de9-809a-446c-aa19-79020a584b4a/volumes"
Oct 30 22:48:00.577434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:00.577400    4581 scope.go:122] "RemoveContainer" containerID="a9c8df3258ecf03eb07abecd88ce82b819f6fcda258c4bd3a400cbfcec048f5b"
Oct 30 22:48:00.589316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:00.589292    4581 scope.go:122] "RemoveContainer" containerID="5e49cee37478604bcee6ab1a86e57dd6cfe275d2a31404625cf70b0fac2975ff"
Oct 30 22:48:01.208761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:01.208722    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps" podUID="ea375e9e-122e-486c-9da0-9e731d72db29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:03.703319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:03.703263    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps" event={"ID":"ea375e9e-122e-486c-9da0-9e731d72db29","Type":"ContainerDied","Data":"55c9d9ad647f2d9b27e1c900e6c20021d768357fb0fe852d6fb7803ceae0cd2c"}
Oct 30 22:48:03.703891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:03.703337    4581 generic.go:356] "Generic (PLEG): container finished" podID="ea375e9e-122e-486c-9da0-9e731d72db29" containerID="55c9d9ad647f2d9b27e1c900e6c20021d768357fb0fe852d6fb7803ceae0cd2c" exitCode=0
Oct 30 22:48:03.906245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:03.892836    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:48:04.113238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.106470    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea375e9e-122e-486c-9da0-9e731d72db29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea375e9e-122e-486c-9da0-9e731d72db29-plugins-config-volume\") pod \"ea375e9e-122e-486c-9da0-9e731d72db29\" (UID: \"ea375e9e-122e-486c-9da0-9e731d72db29\") "
Oct 30 22:48:04.113238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.106690    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea375e9e-122e-486c-9da0-9e731d72db29-kube-api-access-tsjmb\" (UniqueName: \"kubernetes.io/projected/ea375e9e-122e-486c-9da0-9e731d72db29-kube-api-access-tsjmb\") pod \"ea375e9e-122e-486c-9da0-9e731d72db29\" (UID: \"ea375e9e-122e-486c-9da0-9e731d72db29\") "
Oct 30 22:48:04.160341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.160296    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea375e9e-122e-486c-9da0-9e731d72db29-kube-api-access-tsjmb" pod "ea375e9e-122e-486c-9da0-9e731d72db29" (UID: "ea375e9e-122e-486c-9da0-9e731d72db29"). InnerVolumeSpecName "kube-api-access-tsjmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:04.166536 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.166492    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea375e9e-122e-486c-9da0-9e731d72db29-plugins-config-volume" pod "ea375e9e-122e-486c-9da0-9e731d72db29" (UID: "ea375e9e-122e-486c-9da0-9e731d72db29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:04.225758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.225719    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjmb\" (UniqueName: \"kubernetes.io/projected/ea375e9e-122e-486c-9da0-9e731d72db29-kube-api-access-tsjmb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.225758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.225758    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea375e9e-122e-486c-9da0-9e731d72db29-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.301691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.301642    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"]
Oct 30 22:48:04.306197 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.306173    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40229e-tlm24"]
Oct 30 22:48:04.716192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.716161    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"
Oct 30 22:48:04.716192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.716176    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps" event={"ID":"ea375e9e-122e-486c-9da0-9e731d72db29","Type":"ContainerDied","Data":"3e8f45582d67e8812527c35948cc59de6bc3760134cc0e1307b47b00533baa20"}
Oct 30 22:48:04.716725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.716261    4581 scope.go:122] "RemoveContainer" containerID="55c9d9ad647f2d9b27e1c900e6c20021d768357fb0fe852d6fb7803ceae0cd2c"
Oct 30 22:48:04.767006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.766964    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"]
Oct 30 22:48:04.776293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.776253    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40229e-5fb694f79-5glps"]
Oct 30 22:48:04.877250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.877180    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"]
Oct 30 22:48:04.908241 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:04.901877    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-54c7f0-hmkgg"]
Oct 30 22:48:06.100955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:06.100918    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6" path="/var/lib/kubelet/pods/2dcf9887-057f-44d3-9e0f-1bc89c9cdaf6/volumes"
Oct 30 22:48:06.101456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:06.101435    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4afb333-c8b8-468c-953c-66326f449f6c" path="/var/lib/kubelet/pods/a4afb333-c8b8-468c-953c-66326f449f6c/volumes"
Oct 30 22:48:06.101822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:06.101809    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea375e9e-122e-486c-9da0-9e731d72db29" path="/var/lib/kubelet/pods/ea375e9e-122e-486c-9da0-9e731d72db29/volumes"
Oct 30 22:48:10.819939 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:10.819901    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:48:11.953286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:11.953256    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"]
Oct 30 22:48:11.953790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:11.953513    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy" containerID="cri-o://00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d" gracePeriod=30
Oct 30 22:48:12.608019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.607983    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"]
Oct 30 22:48:12.608861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.608841    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ea375e9e-122e-486c-9da0-9e731d72db29" containerName="payload-pre-processing"
Oct 30 22:48:12.609016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.608862    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea375e9e-122e-486c-9da0-9e731d72db29" containerName="payload-pre-processing"
Oct 30 22:48:12.609016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.608880    4581 cpu_manager.go:414] "containerMap: removing container" podUID="78e2cbbd-e9fc-43b5-a683-b3901acb948e" containerName="revoke-keys"
Oct 30 22:48:12.609016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.608887    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e2cbbd-e9fc-43b5-a683-b3901acb948e" containerName="revoke-keys"
Oct 30 22:48:12.609016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.608924    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy"
Oct 30 22:48:12.609016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.608932    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy"
Oct 30 22:48:12.609016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.608944    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy"
Oct 30 22:48:12.609016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.608951    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy"
Oct 30 22:48:12.609016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609016    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" containerName="payload-pre-processing"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609024    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" containerName="payload-pre-processing"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609039    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0f95efa3-f23a-4499-be2a-32ef49ae2205" containerName="revoke-keys"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609046    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f95efa3-f23a-4499-be2a-32ef49ae2205" containerName="revoke-keys"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609238    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609252    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d34bbcd-27ef-45c7-aa3a-bb6011a7fa56" containerName="istio-proxy"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609273    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc54dee-efa4-49ef-84c2-d0d0c3c3ce66" containerName="payload-pre-processing"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609285    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f95efa3-f23a-4499-be2a-32ef49ae2205" containerName="revoke-keys"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609294    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea375e9e-122e-486c-9da0-9e731d72db29" containerName="payload-pre-processing"
Oct 30 22:48:12.609467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.609304    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e2cbbd-e9fc-43b5-a683-b3901acb948e" containerName="revoke-keys"
Oct 30 22:48:12.617732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.617707    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.630077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.630052    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:12.631287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.631263    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"]
Oct 30 22:48:12.631675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.631650    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy" containerID="cri-o://e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c" gracePeriod=30
Oct 30 22:48:12.635499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.635475    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:12.635648 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.635579    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:48:12.637288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.636296    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:12.644067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.644021    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"]
Oct 30 22:48:12.687025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.686990    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.687264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.687030    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.687264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.687066    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwk2\" (UniqueName: \"kubernetes.io/projected/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-kube-api-access-xmwk2\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.842244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.839196    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.842244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.839268    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.842244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.839297    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwk2\" (UniqueName: \"kubernetes.io/projected/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-kube-api-access-xmwk2\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.842244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.840294    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.898300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.878521    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.899857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.899823    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwk2\" (UniqueName: \"kubernetes.io/projected/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-kube-api-access-xmwk2\") pod \"maas-api-revoke-keys-e2e-inf-b-c13db0-7b422\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:12.967531 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:12.967496    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:13.303390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:13.303362    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"]
Oct 30 22:48:13.304588 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:48:13.304557    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebec83d_dcd7_48e4_9ab4_f11daad83cd1.slice/crio-df1c8835a18d1e4faa78c2f11b06b34a0b1f497108f550b512499ea8a9e45268 WatchSource:0}: Error finding container df1c8835a18d1e4faa78c2f11b06b34a0b1f497108f550b512499ea8a9e45268: Status 404 returned error can't find the container with id df1c8835a18d1e4faa78c2f11b06b34a0b1f497108f550b512499ea8a9e45268
Oct 30 22:48:13.880051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:13.879978    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"]
Oct 30 22:48:13.886167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:13.886136    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:13.913011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:13.908425    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"]
Oct 30 22:48:13.918073 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:13.917627    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-bed544-openshift-default-dockercfg-mnfpc\""
Oct 30 22:48:13.928723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:13.928664    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422" event={"ID":"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1","Type":"ContainerStarted","Data":"edbdab631ad4b182765e1ed2d64e928b95368e7e0d207586b1db55c7e1d78cfd"}
Oct 30 22:48:13.928935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:13.928726    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422" event={"ID":"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1","Type":"ContainerStarted","Data":"df1c8835a18d1e4faa78c2f11b06b34a0b1f497108f550b512499ea8a9e45268"}
Oct 30 22:48:14.065421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.046751    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-socket\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.065421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.046861    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-data\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.065421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.046904    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-podinfo\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.065421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.046935    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpsw\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-kube-api-access-kkpsw\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.065421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.047017    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-credential-socket\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.065421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.047079    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-envoy\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.065421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.047254    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-token\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.095601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.084066    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422" podStartSLOduration=2.0840443730000002 podStartE2EDuration="2.084044373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:13.963713765 +0000 UTC m=+2478.002652652" watchObservedRunningTime="2026-10-30 22:48:14.084044373 +0000 UTC m=+2478.122983260"
Oct 30 22:48:14.095601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.086661    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-certs\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.095601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.086743    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istiod-ca-cert\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.188765    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-token\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.188819    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-certs\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.188864    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istiod-ca-cert\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.188946    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-socket\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.189022    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-data\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.189056    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-podinfo\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.189086    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpsw\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-kube-api-access-kkpsw\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.189158    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-credential-socket\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.189369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.189232    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-envoy\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.192249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.190408    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-credential-socket\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.213310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.208079    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-certs\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.213540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.211944    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-socket\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.213669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.212187    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-data\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.214526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.214500    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istiod-ca-cert\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.215169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.215148    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-envoy\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.255036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.254518    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpsw\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-kube-api-access-kkpsw\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.286964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.286887    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-token\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.289201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.289163    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-podinfo\") pod \"e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") " pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.531873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.531564    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:14.866151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.866119    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"]
Oct 30 22:48:14.867593 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:48:14.867551    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91815b9_ea0e_4d30_b6fb_72e9933362cf.slice/crio-713797c74f88ca5b2fb12aa58727ce334de1e605a86a89e28af6d9b5cc13edc5 WatchSource:0}: Error finding container 713797c74f88ca5b2fb12aa58727ce334de1e605a86a89e28af6d9b5cc13edc5: Status 404 returned error can't find the container with id 713797c74f88ca5b2fb12aa58727ce334de1e605a86a89e28af6d9b5cc13edc5
Oct 30 22:48:14.870938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.870896    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:48:14.871035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.870988    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:48:14.871075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.871057    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:48:14.941776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.941743    4581 generic.go:356] "Generic (PLEG): container finished" podID="1ebec83d-dcd7-48e4-9ab4-f11daad83cd1" containerID="edbdab631ad4b182765e1ed2d64e928b95368e7e0d207586b1db55c7e1d78cfd" exitCode=0
Oct 30 22:48:14.941938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.941881    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422" event={"ID":"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1","Type":"ContainerDied","Data":"edbdab631ad4b182765e1ed2d64e928b95368e7e0d207586b1db55c7e1d78cfd"}
Oct 30 22:48:14.943455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:14.943430    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" event={"ID":"f91815b9-ea0e-4d30-b6fb-72e9933362cf","Type":"ContainerStarted","Data":"713797c74f88ca5b2fb12aa58727ce334de1e605a86a89e28af6d9b5cc13edc5"}
Oct 30 22:48:15.126880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.126802    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"]
Oct 30 22:48:15.132537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.132518    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.149424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.149396    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"]
Oct 30 22:48:15.206248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.205793    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.206248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.205858    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpp5\" (UniqueName: \"kubernetes.io/projected/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-kube-api-access-2zpp5\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.206248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.206010    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.308503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.308470    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.308724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.308703    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.308785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.308752    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpp5\" (UniqueName: \"kubernetes.io/projected/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-kube-api-access-2zpp5\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.309328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.309292    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.311357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.311331    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.326135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.326102    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpp5\" (UniqueName: \"kubernetes.io/projected/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-kube-api-access-2zpp5\") pod \"maas-api-revoke-sub-93b944c4-mttjv\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.451202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.451100    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:15.766041 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.766012    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"]
Oct 30 22:48:15.767071 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:48:15.767043    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0e4532_b323_4b58_be3f_fb4b80e1aeea.slice/crio-945de685b332bc65fc2b3d611ed1fd0947053ce868c27a6b3ee25ff3de363705 WatchSource:0}: Error finding container 945de685b332bc65fc2b3d611ed1fd0947053ce868c27a6b3ee25ff3de363705: Status 404 returned error can't find the container with id 945de685b332bc65fc2b3d611ed1fd0947053ce868c27a6b3ee25ff3de363705
Oct 30 22:48:15.956312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.956273    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" event={"ID":"f91815b9-ea0e-4d30-b6fb-72e9933362cf","Type":"ContainerStarted","Data":"4bf414d2c5cc881ac7a528e9e4fa85158c7bc366f69a53ebe6066e5c5a669d49"}
Oct 30 22:48:15.958055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.958025    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv" event={"ID":"6e0e4532-b323-4b58-be3f-fb4b80e1aeea","Type":"ContainerStarted","Data":"945de685b332bc65fc2b3d611ed1fd0947053ce868c27a6b3ee25ff3de363705"}
Oct 30 22:48:15.997058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:15.997004    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" podStartSLOduration=2.996988959 podStartE2EDuration="2.996988959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:15.995287144 +0000 UTC m=+2480.034226029" watchObservedRunningTime="2026-10-30 22:48:15.996988959 +0000 UTC m=+2480.035927844"
Oct 30 22:48:16.186610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.186576    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:16.239826 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.239788    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-kube-api-access-xmwk2\" (UniqueName: \"kubernetes.io/projected/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-kube-api-access-xmwk2\") pod \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") "
Oct 30 22:48:16.240001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.239988    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-xks-ca\") pod \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") "
Oct 30 22:48:16.240063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.240046    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-openshift-ca\") pod \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\" (UID: \"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1\") "
Oct 30 22:48:16.240539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.240504    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-openshift-ca" pod "1ebec83d-dcd7-48e4-9ab4-f11daad83cd1" (UID: "1ebec83d-dcd7-48e4-9ab4-f11daad83cd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:16.240878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.240856    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:16.242646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.242615    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-kube-api-access-xmwk2" pod "1ebec83d-dcd7-48e4-9ab4-f11daad83cd1" (UID: "1ebec83d-dcd7-48e4-9ab4-f11daad83cd1"). InnerVolumeSpecName "kube-api-access-xmwk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:16.243565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.243533    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-xks-ca" pod "1ebec83d-dcd7-48e4-9ab4-f11daad83cd1" (UID: "1ebec83d-dcd7-48e4-9ab4-f11daad83cd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:16.342412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.342331    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmwk2\" (UniqueName: \"kubernetes.io/projected/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-kube-api-access-xmwk2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:16.342412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.342358    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:16.532321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.532285    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:16.538969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.538940    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:16.971443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.971403    4581 generic.go:356] "Generic (PLEG): container finished" podID="6e0e4532-b323-4b58-be3f-fb4b80e1aeea" containerID="dfb481e60b08b1a94a93f4174fdbabc84c0a05f73b7cc67af9d1e20ef97c849d" exitCode=0
Oct 30 22:48:16.971710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.971678    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv" event={"ID":"6e0e4532-b323-4b58-be3f-fb4b80e1aeea","Type":"ContainerDied","Data":"dfb481e60b08b1a94a93f4174fdbabc84c0a05f73b7cc67af9d1e20ef97c849d"}
Oct 30 22:48:16.973336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.973318    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"
Oct 30 22:48:16.973437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.973359    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422" event={"ID":"1ebec83d-dcd7-48e4-9ab4-f11daad83cd1","Type":"ContainerDied","Data":"df1c8835a18d1e4faa78c2f11b06b34a0b1f497108f550b512499ea8a9e45268"}
Oct 30 22:48:16.973437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.973401    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df1c8835a18d1e4faa78c2f11b06b34a0b1f497108f550b512499ea8a9e45268"
Oct 30 22:48:16.973711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.973693    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:16.974593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:16.974563    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:48:17.280389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.280361    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:48:17.356284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356254    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-socket\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.356284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356289    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-envoy\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.356541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356400    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-data\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.356541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356459    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istiod-ca-cert\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.356663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356588    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-token\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-token\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.356663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356621    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-socket" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.356663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356638    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-certs\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.356832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356712    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-podinfo\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.356832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356759    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-credential-socket\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.356832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.356810    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-kube-api-access-qbp9c\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-kube-api-access-qbp9c\") pod \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\" (UID: \"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d\") "
Oct 30 22:48:17.357526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.357494    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-data" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.358053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.358017    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-credential-socket" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.358184 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.358063    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istiod-ca-cert" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:17.358691 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.358665    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-certs" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.360462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.359232    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.360462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.359265    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.360462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.359280    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.360462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.359295    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.360462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.359310    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.362616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.362589    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-envoy" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:17.362616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.362598    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-podinfo" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:17.362738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.362620    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-token" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:17.364409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.364380    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-kube-api-access-qbp9c" pod "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" (UID: "ecdd4c93-87a0-45d3-af7d-a5ad7626d81d"). InnerVolumeSpecName "kube-api-access-qbp9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:17.460848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.460807    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.460848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.460842    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.460848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.460851    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbp9c\" (UniqueName: \"kubernetes.io/projected/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-kube-api-access-qbp9c\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.461093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.460861    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:17.930143 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.930117    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:48:17.987705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.987660    4581 generic.go:356] "Generic (PLEG): container finished" podID="e967c73d-9969-4099-a603-b69060bb3732" containerID="e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c" exitCode=0
Oct 30 22:48:17.987905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.987709    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"
Oct 30 22:48:17.987905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.987783    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" event={"ID":"e967c73d-9969-4099-a603-b69060bb3732","Type":"ContainerDied","Data":"e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c"}
Oct 30 22:48:17.987905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.987859    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn" event={"ID":"e967c73d-9969-4099-a603-b69060bb3732","Type":"ContainerDied","Data":"9a81b5baecb47310990443bb06c804ca9e6b8c55bec11b24712387953529cff5"}
Oct 30 22:48:17.988093 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.987908    4581 scope.go:122] "RemoveContainer" containerID="e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c"
Oct 30 22:48:17.991291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.991180    4581 generic.go:356] "Generic (PLEG): container finished" podID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerID="00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d" exitCode=0
Oct 30 22:48:17.991452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.991426    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" event={"ID":"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d","Type":"ContainerDied","Data":"00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d"}
Oct 30 22:48:17.991551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.991491    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9" event={"ID":"ecdd4c93-87a0-45d3-af7d-a5ad7626d81d","Type":"ContainerDied","Data":"a230ce3ff3e220ed147604316fa8b65da239293a76b2a18f6ece3084c1ea9529"}
Oct 30 22:48:17.991551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:17.991510    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"
Oct 30 22:48:18.019118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.019093    4581 scope.go:122] "RemoveContainer" containerID="e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c"
Oct 30 22:48:18.020354 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:18.020323    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c\": container with ID starting with e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c not found: ID does not exist" containerID="e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c"
Oct 30 22:48:18.020550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.020360    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c"} err="failed to get container status \"e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c\": rpc error: code = NotFound desc = could not find container \"e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c\": container with ID starting with e7a633ca3a0a5265e26e4caac25944d813ce4f437be58c242728c8c2e0dc493c not found: ID does not exist"
Oct 30 22:48:18.020550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.020381    4581 scope.go:122] "RemoveContainer" containerID="00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d"
Oct 30 22:48:18.040987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.040960    4581 scope.go:122] "RemoveContainer" containerID="6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce"
Oct 30 22:48:18.056448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.056421    4581 scope.go:122] "RemoveContainer" containerID="00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d"
Oct 30 22:48:18.056848 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:18.056816    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d\": container with ID starting with 00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d not found: ID does not exist" containerID="00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d"
Oct 30 22:48:18.056935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.056863    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d"} err="failed to get container status \"00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d\": rpc error: code = NotFound desc = could not find container \"00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d\": container with ID starting with 00703871cfcb17e95a7bfc755066da0461a6e60f5abf34d3f375042eb3efea2d not found: ID does not exist"
Oct 30 22:48:18.056935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.056887    4581 scope.go:122] "RemoveContainer" containerID="6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce"
Oct 30 22:48:18.057156 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:18.057132    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce\": container with ID starting with 6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce not found: ID does not exist" containerID="6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce"
Oct 30 22:48:18.057265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.057180    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce"} err="failed to get container status \"6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce\": rpc error: code = NotFound desc = could not find container \"6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce\": container with ID starting with 6defd2c22efa4d1cb9ec27803bb1471d54211e488860d821addd4cc68674a9ce not found: ID does not exist"
Oct 30 22:48:18.062877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.062124    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"]
Oct 30 22:48:18.067916 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.067889    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-40229e-openshift-default-5f54d778f5-zhln9"]
Oct 30 22:48:18.077578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.077552    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-istio-token\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-istio-token\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.077763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.077753    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e967c73d-9969-4099-a603-b69060bb3732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967c73d-9969-4099-a603-b69060bb3732-istiod-ca-cert\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.078153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.078139    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-credential-socket\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.078527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.078515    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-envoy\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.078866 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.078848    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e967c73d-9969-4099-a603-b69060bb3732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967c73d-9969-4099-a603-b69060bb3732-istio-podinfo\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.078989 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.078090    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e967c73d-9969-4099-a603-b69060bb3732-istiod-ca-cert" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:18.079080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.078445    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-credential-socket" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.079172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.079080    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-kube-api-access-gvnk9\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-kube-api-access-gvnk9\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.079422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.079403    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-certs\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.079523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.079500    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-socket\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.079593 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.079569    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-data\") pod \"e967c73d-9969-4099-a603-b69060bb3732\" (UID: \"e967c73d-9969-4099-a603-b69060bb3732\") "
Oct 30 22:48:18.080035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.079873    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-socket" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.080760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.080146    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-data" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.080760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.080489    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.080760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.080510    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.080760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.080526    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e967c73d-9969-4099-a603-b69060bb3732-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.080760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.080541    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.081400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.081338    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-istio-token" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:18.081400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.081391    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-envoy" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.081667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.081641    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-kube-api-access-gvnk9" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "kube-api-access-gvnk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:18.081875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.081850    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-certs" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:18.084079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.084051    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e967c73d-9969-4099-a603-b69060bb3732-istio-podinfo" pod "e967c73d-9969-4099-a603-b69060bb3732" (UID: "e967c73d-9969-4099-a603-b69060bb3732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:18.114934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.114349    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" path="/var/lib/kubelet/pods/ecdd4c93-87a0-45d3-af7d-a5ad7626d81d/volumes"
Oct 30 22:48:18.183082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.183050    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.183082 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.183083    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.183301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.183097    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e967c73d-9969-4099-a603-b69060bb3732-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.183301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.183110    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvnk9\" (UniqueName: \"kubernetes.io/projected/e967c73d-9969-4099-a603-b69060bb3732-kube-api-access-gvnk9\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.183301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.183124    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e967c73d-9969-4099-a603-b69060bb3732-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.209899 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.209868    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:18.283987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.283952    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-openshift-ca\") pod \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") "
Oct 30 22:48:18.284598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.284091    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-kube-api-access-2zpp5\" (UniqueName: \"kubernetes.io/projected/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-kube-api-access-2zpp5\") pod \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") "
Oct 30 22:48:18.284598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.284260    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-xks-ca\") pod \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\" (UID: \"6e0e4532-b323-4b58-be3f-fb4b80e1aeea\") "
Oct 30 22:48:18.284598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.284414    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-openshift-ca" pod "6e0e4532-b323-4b58-be3f-fb4b80e1aeea" (UID: "6e0e4532-b323-4b58-be3f-fb4b80e1aeea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:18.284919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.284896    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.287423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.287396    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-xks-ca" pod "6e0e4532-b323-4b58-be3f-fb4b80e1aeea" (UID: "6e0e4532-b323-4b58-be3f-fb4b80e1aeea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:18.287589 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.287562    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-kube-api-access-2zpp5" pod "6e0e4532-b323-4b58-be3f-fb4b80e1aeea" (UID: "6e0e4532-b323-4b58-be3f-fb4b80e1aeea"). InnerVolumeSpecName "kube-api-access-2zpp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:18.330599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.330564    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"]
Oct 30 22:48:18.335925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.335898    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-54c7f0-openshift-default-79cd69d455-dznhn"]
Oct 30 22:48:18.386181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.386139    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zpp5\" (UniqueName: \"kubernetes.io/projected/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-kube-api-access-2zpp5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:18.386181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:18.386171    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0e4532-b323-4b58-be3f-fb4b80e1aeea-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:19.004321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.004287    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"
Oct 30 22:48:19.004321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.004297    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv" event={"ID":"6e0e4532-b323-4b58-be3f-fb4b80e1aeea","Type":"ContainerDied","Data":"945de685b332bc65fc2b3d611ed1fd0947053ce868c27a6b3ee25ff3de363705"}
Oct 30 22:48:19.004583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.004348    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945de685b332bc65fc2b3d611ed1fd0947053ce868c27a6b3ee25ff3de363705"
Oct 30 22:48:19.506471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.506430    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"]
Oct 30 22:48:19.507398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507371    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1ebec83d-dcd7-48e4-9ab4-f11daad83cd1" containerName="revoke-keys"
Oct 30 22:48:19.507398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507397    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebec83d-dcd7-48e4-9ab4-f11daad83cd1" containerName="revoke-keys"
Oct 30 22:48:19.507590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507419    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy"
Oct 30 22:48:19.507590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507426    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy"
Oct 30 22:48:19.507590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507448    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy"
Oct 30 22:48:19.507590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507455    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy"
Oct 30 22:48:19.507590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507480    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy"
Oct 30 22:48:19.507590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507486    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy"
Oct 30 22:48:19.507590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507509    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6e0e4532-b323-4b58-be3f-fb4b80e1aeea" containerName="revoke-keys"
Oct 30 22:48:19.507590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507515    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0e4532-b323-4b58-be3f-fb4b80e1aeea" containerName="revoke-keys"
Oct 30 22:48:19.507832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507627    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebec83d-dcd7-48e4-9ab4-f11daad83cd1" containerName="revoke-keys"
Oct 30 22:48:19.507832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507635    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy"
Oct 30 22:48:19.507832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507643    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e967c73d-9969-4099-a603-b69060bb3732" containerName="istio-proxy"
Oct 30 22:48:19.507832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.507654    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0e4532-b323-4b58-be3f-fb4b80e1aeea" containerName="revoke-keys"
Oct 30 22:48:19.511609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.511590    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.524763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.524731    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:19.524914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.524895    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:48:19.535547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.535520    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"]
Oct 30 22:48:19.586394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.586236    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:19.586604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.586519    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:19.613979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.613946    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.614160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.613990    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2lp\" (UniqueName: \"kubernetes.io/projected/d5a238af-6739-4e77-aae5-64f6d69bf3ff-kube-api-access-lf2lp\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.614324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.614293    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.716336 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.716298    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.716534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.716361    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2lp\" (UniqueName: \"kubernetes.io/projected/d5a238af-6739-4e77-aae5-64f6d69bf3ff-kube-api-access-lf2lp\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.716534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.716510    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.716955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.716929    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.719161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.719137    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.733696 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.733664    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2lp\" (UniqueName: \"kubernetes.io/projected/d5a238af-6739-4e77-aae5-64f6d69bf3ff-kube-api-access-lf2lp\") pod \"maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:19.827090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:19.826997    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:20.101150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:20.101058    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e967c73d-9969-4099-a603-b69060bb3732" path="/var/lib/kubelet/pods/e967c73d-9969-4099-a603-b69060bb3732/volumes"
Oct 30 22:48:20.140083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:20.140048    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"]
Oct 30 22:48:20.140733 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:48:20.140704    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a238af_6739_4e77_aae5_64f6d69bf3ff.slice/crio-1d732c3b5c33fabb6e958ea710f793c82d945752514dbf19a19dfdfdce768a66 WatchSource:0}: Error finding container 1d732c3b5c33fabb6e958ea710f793c82d945752514dbf19a19dfdfdce768a66: Status 404 returned error can't find the container with id 1d732c3b5c33fabb6e958ea710f793c82d945752514dbf19a19dfdfdce768a66
Oct 30 22:48:21.152930 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:21.152890    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj" event={"ID":"d5a238af-6739-4e77-aae5-64f6d69bf3ff","Type":"ContainerStarted","Data":"9a65836e07578d428651f794c743aaa01f6bb8cf46f95f2329248b61a55ceda8"}
Oct 30 22:48:21.153429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:21.152954    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj" event={"ID":"d5a238af-6739-4e77-aae5-64f6d69bf3ff","Type":"ContainerStarted","Data":"1d732c3b5c33fabb6e958ea710f793c82d945752514dbf19a19dfdfdce768a66"}
Oct 30 22:48:21.193949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:21.193879    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj" podStartSLOduration=2.193860359 podStartE2EDuration="2.193860359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:21.188645508 +0000 UTC m=+2485.227584395" watchObservedRunningTime="2026-10-30 22:48:21.193860359 +0000 UTC m=+2485.232799246"
Oct 30 22:48:22.165151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:22.165109    4581 generic.go:356] "Generic (PLEG): container finished" podID="d5a238af-6739-4e77-aae5-64f6d69bf3ff" containerID="9a65836e07578d428651f794c743aaa01f6bb8cf46f95f2329248b61a55ceda8" exitCode=0
Oct 30 22:48:22.165720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:22.165201    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj" event={"ID":"d5a238af-6739-4e77-aae5-64f6d69bf3ff","Type":"ContainerDied","Data":"9a65836e07578d428651f794c743aaa01f6bb8cf46f95f2329248b61a55ceda8"}
Oct 30 22:48:23.392193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.392163    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:23.483940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.483839    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-openshift-ca\") pod \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") "
Oct 30 22:48:23.483940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.483899    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-xks-ca\") pod \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") "
Oct 30 22:48:23.484175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.484032    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a238af-6739-4e77-aae5-64f6d69bf3ff-kube-api-access-lf2lp\" (UniqueName: \"kubernetes.io/projected/d5a238af-6739-4e77-aae5-64f6d69bf3ff-kube-api-access-lf2lp\") pod \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\" (UID: \"d5a238af-6739-4e77-aae5-64f6d69bf3ff\") "
Oct 30 22:48:23.484663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.484637    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-openshift-ca" pod "d5a238af-6739-4e77-aae5-64f6d69bf3ff" (UID: "d5a238af-6739-4e77-aae5-64f6d69bf3ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:23.491083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.491048    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-xks-ca" pod "d5a238af-6739-4e77-aae5-64f6d69bf3ff" (UID: "d5a238af-6739-4e77-aae5-64f6d69bf3ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:23.491582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.491551    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a238af-6739-4e77-aae5-64f6d69bf3ff-kube-api-access-lf2lp" pod "d5a238af-6739-4e77-aae5-64f6d69bf3ff" (UID: "d5a238af-6739-4e77-aae5-64f6d69bf3ff"). InnerVolumeSpecName "kube-api-access-lf2lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:23.586462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.586428    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2lp\" (UniqueName: \"kubernetes.io/projected/d5a238af-6739-4e77-aae5-64f6d69bf3ff-kube-api-access-lf2lp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:23.586462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.586460    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:23.586685 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.586476    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a238af-6739-4e77-aae5-64f6d69bf3ff-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:23.983960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:23.983922    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"]
Oct 30 22:48:24.232568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:24.225603    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj" event={"ID":"d5a238af-6739-4e77-aae5-64f6d69bf3ff","Type":"ContainerDied","Data":"1d732c3b5c33fabb6e958ea710f793c82d945752514dbf19a19dfdfdce768a66"}
Oct 30 22:48:24.232568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:24.225670    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d732c3b5c33fabb6e958ea710f793c82d945752514dbf19a19dfdfdce768a66"
Oct 30 22:48:24.232568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:24.225741    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"
Oct 30 22:48:28.984714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:28.984662    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn" podUID="a025092a-ad46-4269-950a-21b5b7a0447d" containerName="payload-pre-processing" containerID="cri-o://56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5" gracePeriod=25
Oct 30 22:48:30.098626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:30.098583    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"]
Oct 30 22:48:30.127966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:30.127931    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"]
Oct 30 22:48:32.772242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:32.772179    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"]
Oct 30 22:48:32.781088 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:32.781060    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0bb45-stcmq"]
Oct 30 22:48:33.376526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:33.376479    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn" podUID="a025092a-ad46-4269-950a-21b5b7a0447d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:34.102731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:34.102688    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f95efa3-f23a-4499-be2a-32ef49ae2205" path="/var/lib/kubelet/pods/0f95efa3-f23a-4499-be2a-32ef49ae2205/volumes"
Oct 30 22:48:35.099447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.099391    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8" podUID="5ac035c3-3d22-48ff-bb75-4259779288d6" containerName="payload-processing" containerID="cri-o://862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34" gracePeriod=25
Oct 30 22:48:35.129154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.129102    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s" podUID="de9cc6f1-9ec2-420f-b112-405e97693165" containerName="payload-pre-processing" containerID="cri-o://828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e" gracePeriod=25
Oct 30 22:48:35.846227 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.846170    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"]
Oct 30 22:48:35.846975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.846952    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d5a238af-6739-4e77-aae5-64f6d69bf3ff" containerName="revoke-keys"
Oct 30 22:48:35.847111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.846976    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a238af-6739-4e77-aae5-64f6d69bf3ff" containerName="revoke-keys"
Oct 30 22:48:35.847197 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.847176    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a238af-6739-4e77-aae5-64f6d69bf3ff" containerName="revoke-keys"
Oct 30 22:48:35.847285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.847208    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdd4c93-87a0-45d3-af7d-a5ad7626d81d" containerName="istio-proxy"
Oct 30 22:48:35.852821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.852796    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:35.872624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.872594    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"]
Oct 30 22:48:35.874235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.874191    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:35.874372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.874308    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:48:35.874535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.874512    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:35.874622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.874539    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:35.932534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.932495    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9dqn\" (UniqueName: \"kubernetes.io/projected/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-kube-api-access-d9dqn\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:35.932741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.932720    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:35.932811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:35.932751    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:36.033747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:36.033710    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:36.033747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:36.033749    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:36.034004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:36.033886    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9dqn\" (UniqueName: \"kubernetes.io/projected/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-kube-api-access-d9dqn\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:36.034406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:36.034382    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:36.036493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:36.036460    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:36.050764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:36.050716    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9dqn\" (UniqueName: \"kubernetes.io/projected/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-kube-api-access-d9dqn\") pod \"maas-api-revoke-sub-dc3fc901-s6p68\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:36.168785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:36.168749    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:36.480601 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:48:36.480568    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8ad16c_b773_47fa_9c5d_7f583c5af9a6.slice/crio-dc4e33df869c23ef8b41fdf6793a38be81527b7f696585e7e65081b3e1f09673 WatchSource:0}: Error finding container dc4e33df869c23ef8b41fdf6793a38be81527b7f696585e7e65081b3e1f09673: Status 404 returned error can't find the container with id dc4e33df869c23ef8b41fdf6793a38be81527b7f696585e7e65081b3e1f09673
Oct 30 22:48:36.481807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:36.481781    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"]
Oct 30 22:48:37.312839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:37.312749    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8" podUID="5ac035c3-3d22-48ff-bb75-4259779288d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:37.313263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:37.312756    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s" podUID="de9cc6f1-9ec2-420f-b112-405e97693165" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:37.385790 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:37.385742    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68" event={"ID":"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6","Type":"ContainerStarted","Data":"f09624c14b347807ea0a4245bdd98ed4ee987ba9d5cd238318f8345bc6479f9d"}
Oct 30 22:48:37.385976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:37.385820    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68" event={"ID":"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6","Type":"ContainerStarted","Data":"dc4e33df869c23ef8b41fdf6793a38be81527b7f696585e7e65081b3e1f09673"}
Oct 30 22:48:37.422934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:37.422884    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68" podStartSLOduration=2.422869526 podStartE2EDuration="2.422869526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:37.420172389 +0000 UTC m=+2501.459111286" watchObservedRunningTime="2026-10-30 22:48:37.422869526 +0000 UTC m=+2501.461808407"
Oct 30 22:48:37.737529 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:37.737491    4581 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8ad16c_b773_47fa_9c5d_7f583c5af9a6.slice/crio-conmon-f09624c14b347807ea0a4245bdd98ed4ee987ba9d5cd238318f8345bc6479f9d.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:48:38.400174 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:38.400137    4581 generic.go:356] "Generic (PLEG): container finished" podID="3d8ad16c-b773-47fa-9c5d-7f583c5af9a6" containerID="f09624c14b347807ea0a4245bdd98ed4ee987ba9d5cd238318f8345bc6479f9d" exitCode=0
Oct 30 22:48:38.400663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:38.400264    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68" event={"ID":"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6","Type":"ContainerDied","Data":"f09624c14b347807ea0a4245bdd98ed4ee987ba9d5cd238318f8345bc6479f9d"}
Oct 30 22:48:39.607189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.607157    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:39.707581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.707485    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-kube-api-access-d9dqn\" (UniqueName: \"kubernetes.io/projected/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-kube-api-access-d9dqn\") pod \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") "
Oct 30 22:48:39.707581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.707576    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-openshift-ca\") pod \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") "
Oct 30 22:48:39.707836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.707643    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-xks-ca\") pod \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\" (UID: \"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6\") "
Oct 30 22:48:39.716802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.708298    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-openshift-ca" pod "3d8ad16c-b773-47fa-9c5d-7f583c5af9a6" (UID: "3d8ad16c-b773-47fa-9c5d-7f583c5af9a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:39.716802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.709908    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:39.726710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.726650    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-xks-ca" pod "3d8ad16c-b773-47fa-9c5d-7f583c5af9a6" (UID: "3d8ad16c-b773-47fa-9c5d-7f583c5af9a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:39.728741 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.728713    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-kube-api-access-d9dqn" pod "3d8ad16c-b773-47fa-9c5d-7f583c5af9a6" (UID: "3d8ad16c-b773-47fa-9c5d-7f583c5af9a6"). InnerVolumeSpecName "kube-api-access-d9dqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:39.811933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.811891    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9dqn\" (UniqueName: \"kubernetes.io/projected/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-kube-api-access-d9dqn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:39.811933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.811928    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:39.962387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.962294    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"]
Oct 30 22:48:39.962565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:39.962535    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy" containerID="cri-o://ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7" gracePeriod=30
Oct 30 22:48:40.419059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:40.419026    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"
Oct 30 22:48:40.419059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:40.419034    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68" event={"ID":"3d8ad16c-b773-47fa-9c5d-7f583c5af9a6","Type":"ContainerDied","Data":"dc4e33df869c23ef8b41fdf6793a38be81527b7f696585e7e65081b3e1f09673"}
Oct 30 22:48:40.419352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:40.419111    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc4e33df869c23ef8b41fdf6793a38be81527b7f696585e7e65081b3e1f09673"
Oct 30 22:48:40.819124 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:40.819039    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:48:43.377367 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:43.377327    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn" podUID="a025092a-ad46-4269-950a-21b5b7a0447d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:45.258606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.258533    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:48:45.306092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306047    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc66240-9c5e-43f8-9db0-560736cbe443-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc66240-9c5e-43f8-9db0-560736cbe443-istiod-ca-cert\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.306092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306096    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-certs\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.306092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306160    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-data\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.306092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306491    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-envoy\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.306092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306500    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc66240-9c5e-43f8-9db0-560736cbe443-istiod-ca-cert" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:45.306092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306544    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-kube-api-access-jdvng\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-kube-api-access-jdvng\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306570    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-socket\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306613    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-certs" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306626    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-token\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-token\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306660    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-credential-socket\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.306700    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-podinfo\") pod \"cbc66240-9c5e-43f8-9db0-560736cbe443\" (UID: \"cbc66240-9c5e-43f8-9db0-560736cbe443\") "
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.307057    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-data" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.307321    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc66240-9c5e-43f8-9db0-560736cbe443-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.307342    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.307332    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-credential-socket" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.307510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.307356    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.308789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.308765    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-socket" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.309377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.309354    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-envoy" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:45.310098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.310072    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-kube-api-access-jdvng" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "kube-api-access-jdvng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:45.310396 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.310372    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-podinfo" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:45.310983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.310958    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-token" pod "cbc66240-9c5e-43f8-9db0-560736cbe443" (UID: "cbc66240-9c5e-43f8-9db0-560736cbe443"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:45.408674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.408642    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.408674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.408672    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvng\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-kube-api-access-jdvng\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.408882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.408683    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.408882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.408692    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.408882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.408700    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc66240-9c5e-43f8-9db0-560736cbe443-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.408882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.408708    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc66240-9c5e-43f8-9db0-560736cbe443-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:45.470820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.470780    4581 generic.go:356] "Generic (PLEG): container finished" podID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerID="ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7" exitCode=0
Oct 30 22:48:45.471020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.470853    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" event={"ID":"cbc66240-9c5e-43f8-9db0-560736cbe443","Type":"ContainerDied","Data":"ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7"}
Oct 30 22:48:45.471020 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.470910    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"
Oct 30 22:48:45.471144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.470923    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q" event={"ID":"cbc66240-9c5e-43f8-9db0-560736cbe443","Type":"ContainerDied","Data":"e83ad8432fef9f00f1e16839b6c52d156f50258bbb51a39d72ba9185f43b2419"}
Oct 30 22:48:45.471144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.470935    4581 scope.go:122] "RemoveContainer" containerID="ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7"
Oct 30 22:48:45.486892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.486871    4581 scope.go:122] "RemoveContainer" containerID="97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6"
Oct 30 22:48:45.538177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.532557    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"]
Oct 30 22:48:45.541321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.541287    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f0bb45-openshift-default-6b64cff887-s9c4q"]
Oct 30 22:48:45.550029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.550006    4581 scope.go:122] "RemoveContainer" containerID="ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7"
Oct 30 22:48:45.552461 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:45.552431    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7\": container with ID starting with ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7 not found: ID does not exist" containerID="ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7"
Oct 30 22:48:45.552584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.552469    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7"} err="failed to get container status \"ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7\": rpc error: code = NotFound desc = could not find container \"ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7\": container with ID starting with ca25f61467841ee118914fb2d4c3ee3404aafc7b977bc522fd75af97d25ca6f7 not found: ID does not exist"
Oct 30 22:48:45.552584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.552489    4581 scope.go:122] "RemoveContainer" containerID="97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6"
Oct 30 22:48:45.552828 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:45.552801    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6\": container with ID starting with 97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6 not found: ID does not exist" containerID="97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6"
Oct 30 22:48:45.552912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:45.552837    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6"} err="failed to get container status \"97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6\": rpc error: code = NotFound desc = could not find container \"97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6\": container with ID starting with 97fd75be6472b976592debe8daa4bbe4382a496cf9eb5bd283066f72f65669a6 not found: ID does not exist"
Oct 30 22:48:46.101560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:46.101520    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" path="/var/lib/kubelet/pods/cbc66240-9c5e-43f8-9db0-560736cbe443/volumes"
Oct 30 22:48:47.342244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:47.341362    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s" podUID="de9cc6f1-9ec2-420f-b112-405e97693165" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:47.347243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:47.342935    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8" podUID="5ac035c3-3d22-48ff-bb75-4259779288d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:49.336561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.336526    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:48:49.445505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.445470    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a025092a-ad46-4269-950a-21b5b7a0447d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a025092a-ad46-4269-950a-21b5b7a0447d-plugins-config-volume\") pod \"a025092a-ad46-4269-950a-21b5b7a0447d\" (UID: \"a025092a-ad46-4269-950a-21b5b7a0447d\") "
Oct 30 22:48:49.445710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.445637    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a025092a-ad46-4269-950a-21b5b7a0447d-kube-api-access-t9zkb\" (UniqueName: \"kubernetes.io/projected/a025092a-ad46-4269-950a-21b5b7a0447d-kube-api-access-t9zkb\") pod \"a025092a-ad46-4269-950a-21b5b7a0447d\" (UID: \"a025092a-ad46-4269-950a-21b5b7a0447d\") "
Oct 30 22:48:49.445882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.445856    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a025092a-ad46-4269-950a-21b5b7a0447d-plugins-config-volume" pod "a025092a-ad46-4269-950a-21b5b7a0447d" (UID: "a025092a-ad46-4269-950a-21b5b7a0447d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:49.446135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.446115    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a025092a-ad46-4269-950a-21b5b7a0447d-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:49.448133 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.448091    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a025092a-ad46-4269-950a-21b5b7a0447d-kube-api-access-t9zkb" pod "a025092a-ad46-4269-950a-21b5b7a0447d" (UID: "a025092a-ad46-4269-950a-21b5b7a0447d"). InnerVolumeSpecName "kube-api-access-t9zkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:49.547369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.547325    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zkb\" (UniqueName: \"kubernetes.io/projected/a025092a-ad46-4269-950a-21b5b7a0447d-kube-api-access-t9zkb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:49.604608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.604566    4581 generic.go:356] "Generic (PLEG): container finished" podID="a025092a-ad46-4269-950a-21b5b7a0447d" containerID="56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5" exitCode=0
Oct 30 22:48:49.604786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.604615    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"
Oct 30 22:48:49.604786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.604660    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn" event={"ID":"a025092a-ad46-4269-950a-21b5b7a0447d","Type":"ContainerDied","Data":"56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5"}
Oct 30 22:48:49.604786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.604716    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn" event={"ID":"a025092a-ad46-4269-950a-21b5b7a0447d","Type":"ContainerDied","Data":"f4d26e388dfa3e84c63489026bfa7a6e368017192e008c36c5baf3d9c65a92f4"}
Oct 30 22:48:49.604786 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.604739    4581 scope.go:122] "RemoveContainer" containerID="56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5"
Oct 30 22:48:49.618064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.618043    4581 scope.go:122] "RemoveContainer" containerID="56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5"
Oct 30 22:48:49.618416 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:49.618393    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5\": container with ID starting with 56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5 not found: ID does not exist" containerID="56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5"
Oct 30 22:48:49.618530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.618424    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5"} err="failed to get container status \"56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5\": rpc error: code = NotFound desc = could not find container \"56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5\": container with ID starting with 56fb3bf1fe4a341bc0f94a9b95827b77a55e20442821d49bc2b4a04a7935d2f5 not found: ID does not exist"
Oct 30 22:48:49.656857 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.656818    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"]
Oct 30 22:48:49.661595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:49.661561    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c13db0-bcdbd4c9f-tqwjn"]
Oct 30 22:48:50.097736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:50.097649    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a025092a-ad46-4269-950a-21b5b7a0447d" path="/var/lib/kubelet/pods/a025092a-ad46-4269-950a-21b5b7a0447d/volumes"
Oct 30 22:48:55.184249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.184172    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"]
Oct 30 22:48:55.193094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.193071    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c13db0-7b422"]
Oct 30 22:48:55.482895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.482871    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:48:55.487980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.487956    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:48:55.510464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.510440    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9cc6f1-9ec2-420f-b112-405e97693165-kube-api-access-9965l\" (UniqueName: \"kubernetes.io/projected/de9cc6f1-9ec2-420f-b112-405e97693165-kube-api-access-9965l\") pod \"de9cc6f1-9ec2-420f-b112-405e97693165\" (UID: \"de9cc6f1-9ec2-420f-b112-405e97693165\") "
Oct 30 22:48:55.510598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.510558    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9cc6f1-9ec2-420f-b112-405e97693165-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9cc6f1-9ec2-420f-b112-405e97693165-plugins-config-volume\") pod \"de9cc6f1-9ec2-420f-b112-405e97693165\" (UID: \"de9cc6f1-9ec2-420f-b112-405e97693165\") "
Oct 30 22:48:55.510598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.510577    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac035c3-3d22-48ff-bb75-4259779288d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac035c3-3d22-48ff-bb75-4259779288d6-plugins-config-volume\") pod \"5ac035c3-3d22-48ff-bb75-4259779288d6\" (UID: \"5ac035c3-3d22-48ff-bb75-4259779288d6\") "
Oct 30 22:48:55.510720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.510617    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac035c3-3d22-48ff-bb75-4259779288d6-kube-api-access-gd77n\" (UniqueName: \"kubernetes.io/projected/5ac035c3-3d22-48ff-bb75-4259779288d6-kube-api-access-gd77n\") pod \"5ac035c3-3d22-48ff-bb75-4259779288d6\" (UID: \"5ac035c3-3d22-48ff-bb75-4259779288d6\") "
Oct 30 22:48:55.510981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.510949    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9cc6f1-9ec2-420f-b112-405e97693165-plugins-config-volume" pod "de9cc6f1-9ec2-420f-b112-405e97693165" (UID: "de9cc6f1-9ec2-420f-b112-405e97693165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:55.511097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.510996    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac035c3-3d22-48ff-bb75-4259779288d6-plugins-config-volume" pod "5ac035c3-3d22-48ff-bb75-4259779288d6" (UID: "5ac035c3-3d22-48ff-bb75-4259779288d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:55.511403 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.511385    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9cc6f1-9ec2-420f-b112-405e97693165-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:55.511465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.511411    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac035c3-3d22-48ff-bb75-4259779288d6-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:55.512809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.512789    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9cc6f1-9ec2-420f-b112-405e97693165-kube-api-access-9965l" pod "de9cc6f1-9ec2-420f-b112-405e97693165" (UID: "de9cc6f1-9ec2-420f-b112-405e97693165"). InnerVolumeSpecName "kube-api-access-9965l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:55.512893 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.512869    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac035c3-3d22-48ff-bb75-4259779288d6-kube-api-access-gd77n" pod "5ac035c3-3d22-48ff-bb75-4259779288d6" (UID: "5ac035c3-3d22-48ff-bb75-4259779288d6"). InnerVolumeSpecName "kube-api-access-gd77n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:55.612291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.612258    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd77n\" (UniqueName: \"kubernetes.io/projected/5ac035c3-3d22-48ff-bb75-4259779288d6-kube-api-access-gd77n\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:55.612291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.612285    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9965l\" (UniqueName: \"kubernetes.io/projected/de9cc6f1-9ec2-420f-b112-405e97693165-kube-api-access-9965l\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:48:55.664002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.663967    4581 generic.go:356] "Generic (PLEG): container finished" podID="5ac035c3-3d22-48ff-bb75-4259779288d6" containerID="862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34" exitCode=0
Oct 30 22:48:55.664242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.664021    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"
Oct 30 22:48:55.664242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.664060    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8" event={"ID":"5ac035c3-3d22-48ff-bb75-4259779288d6","Type":"ContainerDied","Data":"862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34"}
Oct 30 22:48:55.664242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.664118    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8" event={"ID":"5ac035c3-3d22-48ff-bb75-4259779288d6","Type":"ContainerDied","Data":"185e8d1171e3729ffe58fca566829efe362c2ca637461c37ae11bd9b2bfe4681"}
Oct 30 22:48:55.664242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.664149    4581 scope.go:122] "RemoveContainer" containerID="862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34"
Oct 30 22:48:55.666514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.666482    4581 generic.go:356] "Generic (PLEG): container finished" podID="de9cc6f1-9ec2-420f-b112-405e97693165" containerID="828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e" exitCode=0
Oct 30 22:48:55.666680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.666654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s" event={"ID":"de9cc6f1-9ec2-420f-b112-405e97693165","Type":"ContainerDied","Data":"828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e"}
Oct 30 22:48:55.666757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.666695    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s" event={"ID":"de9cc6f1-9ec2-420f-b112-405e97693165","Type":"ContainerDied","Data":"e01668a050f79603d95a7d7ede4e3990feaa9552349a95c55639c401722bb0f8"}
Oct 30 22:48:55.667120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.667100    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"
Oct 30 22:48:55.678959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.678932    4581 scope.go:122] "RemoveContainer" containerID="862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34"
Oct 30 22:48:55.679360 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:55.679337    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34\": container with ID starting with 862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34 not found: ID does not exist" containerID="862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34"
Oct 30 22:48:55.679454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.679369    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34"} err="failed to get container status \"862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34\": rpc error: code = NotFound desc = could not find container \"862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34\": container with ID starting with 862218bf63d35ec0d0895ce9eb05d73293856e3afa280cf23a9228d6d220ae34 not found: ID does not exist"
Oct 30 22:48:55.679454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.679386    4581 scope.go:122] "RemoveContainer" containerID="828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e"
Oct 30 22:48:55.691602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.691581    4581 scope.go:122] "RemoveContainer" containerID="828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e"
Oct 30 22:48:55.691918 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:48:55.691894    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e\": container with ID starting with 828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e not found: ID does not exist" containerID="828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e"
Oct 30 22:48:55.692000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.691928    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e"} err="failed to get container status \"828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e\": rpc error: code = NotFound desc = could not find container \"828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e\": container with ID starting with 828119f19bd4e07a2c0c80575e112fb7b3d9d4375abc2ac29a8aeff3ee199a7e not found: ID does not exist"
Oct 30 22:48:55.713061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.713030    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"]
Oct 30 22:48:55.718195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.718174    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e0edc-c7974cff5-rr5v8"]
Oct 30 22:48:55.757071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.757035    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"]
Oct 30 22:48:55.762085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:55.762054    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e0edc-6cb4f7d887-wn55s"]
Oct 30 22:48:56.099399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:56.099315    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebec83d-dcd7-48e4-9ab4-f11daad83cd1" path="/var/lib/kubelet/pods/1ebec83d-dcd7-48e4-9ab4-f11daad83cd1/volumes"
Oct 30 22:48:56.099912 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:56.099890    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac035c3-3d22-48ff-bb75-4259779288d6" path="/var/lib/kubelet/pods/5ac035c3-3d22-48ff-bb75-4259779288d6/volumes"
Oct 30 22:48:56.100380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:48:56.100360    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9cc6f1-9ec2-420f-b112-405e97693165" path="/var/lib/kubelet/pods/de9cc6f1-9ec2-420f-b112-405e97693165/volumes"
Oct 30 22:49:00.701442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:00.701403    4581 scope.go:122] "RemoveContainer" containerID="eb9d1d544a66c24e7672a6456980702cf62a8ed1cfe9f283eb7ae6ceb86b554d"
Oct 30 22:49:00.907452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:00.907420    4581 scope.go:122] "RemoveContainer" containerID="18a6ad29f49652b3d1bea6254f90c4b94a4c63b48d9d5e297e13f4478d13c0c0"
Oct 30 22:49:00.932003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:00.931939    4581 scope.go:122] "RemoveContainer" containerID="bd37296461c3cca1a64b6ef0ee2e31c669444374ff8a406aa7ac9dc72840c805"
Oct 30 22:49:00.944476 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:00.944457    4581 scope.go:122] "RemoveContainer" containerID="686ea8d3b3343ae08cd6eb5897f9117b4f6f340dc26a3a2efb997868c704cfde"
Oct 30 22:49:01.039488 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:01.039463    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"]
Oct 30 22:49:01.047487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:01.047463    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e0edc-bkphj"]
Oct 30 22:49:02.099043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:02.099006    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a238af-6739-4e77-aae5-64f6d69bf3ff" path="/var/lib/kubelet/pods/d5a238af-6739-4e77-aae5-64f6d69bf3ff/volumes"
Oct 30 22:49:02.419054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:02.419024    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"]
Oct 30 22:49:02.419347 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:02.419321    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy" containerID="cri-o://11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc" gracePeriod=30
Oct 30 22:49:05.643430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:05.643390    4581 patch_prober.go:29] interesting pod/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:49:05.643858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:05.643459    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:49:07.711386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.711355    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:49:07.871299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871265    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-kube-api-access-qvbx2\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-kube-api-access-qvbx2\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.871505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871306    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-certs\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.871505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871445    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-credential-socket\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.871505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871485    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-socket\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.871690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871524    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-podinfo\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.871690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871548    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-envoy\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.871690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871648    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istiod-ca-cert\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.871690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871661    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-certs" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:07.871919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871721    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-socket" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:07.871919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871743    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-credential-socket" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:07.871919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871849    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-token\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-token\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.871919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.871913    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-data\") pod \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\" (UID: \"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db\") "
Oct 30 22:49:07.872360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.872046    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istiod-ca-cert" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:07.872360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.872191    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-data" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:07.872798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.872773    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.872900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.872803    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.872900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.872819    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.872900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.872834    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.872900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.872849    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.873954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.873928    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-kube-api-access-qvbx2" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "kube-api-access-qvbx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:07.874211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.874191    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-envoy" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:07.874329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.874279    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-podinfo" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:07.874374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.874331    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-token" pod "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" (UID: "9a9bdb1e-eaf9-410d-9fe5-3337138ca4db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:07.957832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.957732    4581 generic.go:356] "Generic (PLEG): container finished" podID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerID="11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc" exitCode=0
Oct 30 22:49:07.957832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.957793    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"
Oct 30 22:49:07.958077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.957822    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" event={"ID":"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db","Type":"ContainerDied","Data":"11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc"}
Oct 30 22:49:07.958077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.957881    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq" event={"ID":"9a9bdb1e-eaf9-410d-9fe5-3337138ca4db","Type":"ContainerDied","Data":"4b81eb896bb8dd3b3b3a45c993b3bf169639ad723a985e4c91fb9b211c79c739"}
Oct 30 22:49:07.958077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.957896    4581 scope.go:122] "RemoveContainer" containerID="11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc"
Oct 30 22:49:07.970061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.970038    4581 scope.go:122] "RemoveContainer" containerID="c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d"
Oct 30 22:49:07.973683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.973656    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbx2\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-kube-api-access-qvbx2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.973683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.973685    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.973839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.973699    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.973839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.973712    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:07.988824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.988798    4581 scope.go:122] "RemoveContainer" containerID="11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc"
Oct 30 22:49:07.989118 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:07.989098    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc\": container with ID starting with 11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc not found: ID does not exist" containerID="11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc"
Oct 30 22:49:07.989205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.989130    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc"} err="failed to get container status \"11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc\": rpc error: code = NotFound desc = could not find container \"11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc\": container with ID starting with 11a3c5645da3e9b42be2501e586757ee0075e5a682b320cd7c05daa99265b4fc not found: ID does not exist"
Oct 30 22:49:07.989205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.989154    4581 scope.go:122] "RemoveContainer" containerID="c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d"
Oct 30 22:49:07.989442 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:07.989420    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d\": container with ID starting with c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d not found: ID does not exist" containerID="c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d"
Oct 30 22:49:07.989499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:07.989451    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d"} err="failed to get container status \"c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d\": rpc error: code = NotFound desc = could not find container \"c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d\": container with ID starting with c8e26556b961c0a40c6f0f9772ad0b4f1d6a81778033bdb9325aa1fd009b709d not found: ID does not exist"
Oct 30 22:49:08.011599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.011568    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"]
Oct 30 22:49:08.016511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.016469    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c13db0-openshift-default-f894f747-cbtvq"]
Oct 30 22:49:08.098189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.098156    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" path="/var/lib/kubelet/pods/9a9bdb1e-eaf9-410d-9fe5-3337138ca4db/volumes"
Oct 30 22:49:08.409598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.409550    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"]
Oct 30 22:49:08.410089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.409921    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy" containerID="cri-o://e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25" gracePeriod=30
Oct 30 22:49:08.969445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.969388    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"]
Oct 30 22:49:08.970333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970312    4581 cpu_manager.go:414] "containerMap: removing container" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy"
Oct 30 22:49:08.970333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970334    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970368    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5ac035c3-3d22-48ff-bb75-4259779288d6" containerName="payload-processing"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970376    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac035c3-3d22-48ff-bb75-4259779288d6" containerName="payload-processing"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970404    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970414    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970422    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3d8ad16c-b773-47fa-9c5d-7f583c5af9a6" containerName="revoke-keys"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970429    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8ad16c-b773-47fa-9c5d-7f583c5af9a6" containerName="revoke-keys"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970452    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a025092a-ad46-4269-950a-21b5b7a0447d" containerName="payload-pre-processing"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970459    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a025092a-ad46-4269-950a-21b5b7a0447d" containerName="payload-pre-processing"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970474    4581 cpu_manager.go:414] "containerMap: removing container" podUID="de9cc6f1-9ec2-420f-b112-405e97693165" containerName="payload-pre-processing"
Oct 30 22:49:08.970501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970482    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9cc6f1-9ec2-420f-b112-405e97693165" containerName="payload-pre-processing"
Oct 30 22:49:08.970855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970628    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy"
Oct 30 22:49:08.970855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970641    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8ad16c-b773-47fa-9c5d-7f583c5af9a6" containerName="revoke-keys"
Oct 30 22:49:08.970855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970652    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy"
Oct 30 22:49:08.970855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970660    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy"
Oct 30 22:49:08.970855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970668    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a025092a-ad46-4269-950a-21b5b7a0447d" containerName="payload-pre-processing"
Oct 30 22:49:08.970855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970677    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9cc6f1-9ec2-420f-b112-405e97693165" containerName="payload-pre-processing"
Oct 30 22:49:08.970855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970690    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy"
Oct 30 22:49:08.970855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.970702    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac035c3-3d22-48ff-bb75-4259779288d6" containerName="payload-processing"
Oct 30 22:49:08.976006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.975987    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:08.993738 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.993699    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"]
Oct 30 22:49:08.996668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.996620    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:08.996668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.996620    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:08.996668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.996620    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:08.998542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:08.998522    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:49:09.096287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.096242    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.096497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.096444    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlhl\" (UniqueName: \"kubernetes.io/projected/818ad606-0326-4722-b613-f39cbbdc392a-kube-api-access-nnlhl\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.096693 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.096669    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.198348 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.198314    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.198575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.198440    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.198575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.198554    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlhl\" (UniqueName: \"kubernetes.io/projected/818ad606-0326-4722-b613-f39cbbdc392a-kube-api-access-nnlhl\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.199160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.199130    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.201114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.201088    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.216314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.216282    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlhl\" (UniqueName: \"kubernetes.io/projected/818ad606-0326-4722-b613-f39cbbdc392a-kube-api-access-nnlhl\") pod \"maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.292519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.292433    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:09.601114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.601082    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"]
Oct 30 22:49:09.601937 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:49:09.601911    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818ad606_0326_4722_b613_f39cbbdc392a.slice/crio-60b8e5877e6025387518561861f61335a5a1c94cc3bdf0ddc7b9b09a63016896 WatchSource:0}: Error finding container 60b8e5877e6025387518561861f61335a5a1c94cc3bdf0ddc7b9b09a63016896: Status 404 returned error can't find the container with id 60b8e5877e6025387518561861f61335a5a1c94cc3bdf0ddc7b9b09a63016896
Oct 30 22:49:09.979570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.979523    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq" event={"ID":"818ad606-0326-4722-b613-f39cbbdc392a","Type":"ContainerStarted","Data":"e53a8964f14588a3e0fe5f262da134b9ade8ff32728ea007435c1b261ad23edc"}
Oct 30 22:49:09.979984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:09.979629    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq" event={"ID":"818ad606-0326-4722-b613-f39cbbdc392a","Type":"ContainerStarted","Data":"60b8e5877e6025387518561861f61335a5a1c94cc3bdf0ddc7b9b09a63016896"}
Oct 30 22:49:10.017124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:10.017068    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq" podStartSLOduration=2.017053543 podStartE2EDuration="2.017053543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:10.013610066 +0000 UTC m=+2534.052548952" watchObservedRunningTime="2026-10-30 22:49:10.017053543 +0000 UTC m=+2534.055992495"
Oct 30 22:49:10.819206 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:10.819161    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:49:10.990076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:10.990033    4581 generic.go:356] "Generic (PLEG): container finished" podID="818ad606-0326-4722-b613-f39cbbdc392a" containerID="e53a8964f14588a3e0fe5f262da134b9ade8ff32728ea007435c1b261ad23edc" exitCode=0
Oct 30 22:49:10.990583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:10.990113    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq" event={"ID":"818ad606-0326-4722-b613-f39cbbdc392a","Type":"ContainerDied","Data":"e53a8964f14588a3e0fe5f262da134b9ade8ff32728ea007435c1b261ad23edc"}
Oct 30 22:49:12.187293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.187251    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"]
Oct 30 22:49:12.188820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.188795    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy"
Oct 30 22:49:12.188967 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.188955    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9bdb1e-eaf9-410d-9fe5-3337138ca4db" containerName="istio-proxy"
Oct 30 22:49:12.189110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.189100    4581 cpu_manager.go:414] "containerMap: removing container" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy"
Oct 30 22:49:12.189193 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.189183    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc66240-9c5e-43f8-9db0-560736cbe443" containerName="istio-proxy"
Oct 30 22:49:12.193799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.193778    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.212576 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.212550    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"]
Oct 30 22:49:12.273098 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.273073    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:12.353724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.353692    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.353951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.353803    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxk7\" (UniqueName: \"kubernetes.io/projected/f79fd7df-a950-4f68-868c-e8af827e830d-kube-api-access-wpxk7\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.354061 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.354041    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.455467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.455429    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-openshift-ca\") pod \"818ad606-0326-4722-b613-f39cbbdc392a\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") "
Oct 30 22:49:12.455656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.455563    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-xks-ca\") pod \"818ad606-0326-4722-b613-f39cbbdc392a\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") "
Oct 30 22:49:12.455656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.455597    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818ad606-0326-4722-b613-f39cbbdc392a-kube-api-access-nnlhl\" (UniqueName: \"kubernetes.io/projected/818ad606-0326-4722-b613-f39cbbdc392a-kube-api-access-nnlhl\") pod \"818ad606-0326-4722-b613-f39cbbdc392a\" (UID: \"818ad606-0326-4722-b613-f39cbbdc392a\") "
Oct 30 22:49:12.455897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.455868    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-openshift-ca" pod "818ad606-0326-4722-b613-f39cbbdc392a" (UID: "818ad606-0326-4722-b613-f39cbbdc392a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:12.456203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.456178    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.456533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.456285    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.456533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.456384    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxk7\" (UniqueName: \"kubernetes.io/projected/f79fd7df-a950-4f68-868c-e8af827e830d-kube-api-access-wpxk7\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.456533 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.456517    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:12.457731 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.457575    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.458439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.458410    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-xks-ca" pod "818ad606-0326-4722-b613-f39cbbdc392a" (UID: "818ad606-0326-4722-b613-f39cbbdc392a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:12.458581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.458547    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818ad606-0326-4722-b613-f39cbbdc392a-kube-api-access-nnlhl" pod "818ad606-0326-4722-b613-f39cbbdc392a" (UID: "818ad606-0326-4722-b613-f39cbbdc392a"). InnerVolumeSpecName "kube-api-access-nnlhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:12.459260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.459200    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.477626 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.477598    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxk7\" (UniqueName: \"kubernetes.io/projected/f79fd7df-a950-4f68-868c-e8af827e830d-kube-api-access-wpxk7\") pod \"maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.516391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.516361    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:12.558305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.558271    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818ad606-0326-4722-b613-f39cbbdc392a-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:12.558305 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.558306    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnlhl\" (UniqueName: \"kubernetes.io/projected/818ad606-0326-4722-b613-f39cbbdc392a-kube-api-access-nnlhl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:12.828281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:12.828256    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"]
Oct 30 22:49:12.830378 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:49:12.830349    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79fd7df_a950_4f68_868c_e8af827e830d.slice/crio-f1f69da369f472a4c89e8142f5b714ecaeba39d38ce7dfc249584bdde14edfd5 WatchSource:0}: Error finding container f1f69da369f472a4c89e8142f5b714ecaeba39d38ce7dfc249584bdde14edfd5: Status 404 returned error can't find the container with id f1f69da369f472a4c89e8142f5b714ecaeba39d38ce7dfc249584bdde14edfd5
Oct 30 22:49:13.081460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.081377    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"
Oct 30 22:49:13.081460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.081387    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq" event={"ID":"818ad606-0326-4722-b613-f39cbbdc392a","Type":"ContainerDied","Data":"60b8e5877e6025387518561861f61335a5a1c94cc3bdf0ddc7b9b09a63016896"}
Oct 30 22:49:13.081460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.081440    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b8e5877e6025387518561861f61335a5a1c94cc3bdf0ddc7b9b09a63016896"
Oct 30 22:49:13.083311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.083265    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl" event={"ID":"f79fd7df-a950-4f68-868c-e8af827e830d","Type":"ContainerStarted","Data":"f1f69da369f472a4c89e8142f5b714ecaeba39d38ce7dfc249584bdde14edfd5"}
Oct 30 22:49:13.724171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.724142    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:49:13.798571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.798532    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-certs\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.798770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.798625    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-credential-socket\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.798770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.798658    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-istio-token\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-istio-token\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.798896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.798799    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-kube-api-access-2sq9h\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-kube-api-access-2sq9h\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.798896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.798851    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7a61397-afcd-4557-80c6-de16c4023d99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a61397-afcd-4557-80c6-de16c4023d99-istio-podinfo\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.798997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.798904    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-socket\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.798997 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.798947    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-envoy\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.799117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.799102    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-data\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.799171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.799141    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a61397-afcd-4557-80c6-de16c4023d99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a61397-afcd-4557-80c6-de16c4023d99-istiod-ca-cert\") pod \"f7a61397-afcd-4557-80c6-de16c4023d99\" (UID: \"f7a61397-afcd-4557-80c6-de16c4023d99\") "
Oct 30 22:49:13.800821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.800744    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a61397-afcd-4557-80c6-de16c4023d99-istiod-ca-cert" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:13.800959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.800856    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-certs" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.800959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.800868    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-socket" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.801498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.801278    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-credential-socket" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.801933 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.801807    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-data" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.806303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.803606    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-istio-token" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:13.806303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.804337    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7a61397-afcd-4557-80c6-de16c4023d99-istio-podinfo" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:13.806303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.805352    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-kube-api-access-2sq9h" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "kube-api-access-2sq9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:13.806715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.806689    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-envoy" pod "f7a61397-afcd-4557-80c6-de16c4023d99" (UID: "f7a61397-afcd-4557-80c6-de16c4023d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.901196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901154    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.901196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901198    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.901391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901259    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sq9h\" (UniqueName: \"kubernetes.io/projected/f7a61397-afcd-4557-80c6-de16c4023d99-kube-api-access-2sq9h\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.901391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901276    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a61397-afcd-4557-80c6-de16c4023d99-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.901391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901290    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.901391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901303    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.901391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901315    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.901391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901327    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a61397-afcd-4557-80c6-de16c4023d99-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.901391 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:13.901341    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a61397-afcd-4557-80c6-de16c4023d99-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:14.096822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.096741    4581 generic.go:356] "Generic (PLEG): container finished" podID="f7a61397-afcd-4557-80c6-de16c4023d99" containerID="e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25" exitCode=0
Oct 30 22:49:14.096971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.096876    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"
Oct 30 22:49:14.099031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.098999    4581 generic.go:356] "Generic (PLEG): container finished" podID="f79fd7df-a950-4f68-868c-e8af827e830d" containerID="6a40ed284bfc79983f7a5db1808d89c70a2c304cc31815cae603c0ad6c2b09c4" exitCode=0
Oct 30 22:49:14.099828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.099805    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" event={"ID":"f7a61397-afcd-4557-80c6-de16c4023d99","Type":"ContainerDied","Data":"e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25"}
Oct 30 22:49:14.099913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.099850    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv" event={"ID":"f7a61397-afcd-4557-80c6-de16c4023d99","Type":"ContainerDied","Data":"dd0b9fdc9872cc6c9a97a5b5695d36a7ae040e6a489ec3f54d195785cc1d90bc"}
Oct 30 22:49:14.099913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.099876    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl" event={"ID":"f79fd7df-a950-4f68-868c-e8af827e830d","Type":"ContainerDied","Data":"6a40ed284bfc79983f7a5db1808d89c70a2c304cc31815cae603c0ad6c2b09c4"}
Oct 30 22:49:14.099913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.099908    4581 scope.go:122] "RemoveContainer" containerID="e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25"
Oct 30 22:49:14.113689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.113673    4581 scope.go:122] "RemoveContainer" containerID="94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411"
Oct 30 22:49:14.126894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.126874    4581 scope.go:122] "RemoveContainer" containerID="e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25"
Oct 30 22:49:14.127259 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:14.127230    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25\": container with ID starting with e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25 not found: ID does not exist" containerID="e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25"
Oct 30 22:49:14.127328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.127272    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25"} err="failed to get container status \"e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25\": rpc error: code = NotFound desc = could not find container \"e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25\": container with ID starting with e3f2ed9935838a2b2a9ee5d039d4a162748b7b5b6a25d52ef14a8c5c240c0e25 not found: ID does not exist"
Oct 30 22:49:14.127328 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.127296    4581 scope.go:122] "RemoveContainer" containerID="94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411"
Oct 30 22:49:14.127619 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:14.127599    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411\": container with ID starting with 94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411 not found: ID does not exist" containerID="94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411"
Oct 30 22:49:14.127689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.127624    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411"} err="failed to get container status \"94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411\": rpc error: code = NotFound desc = could not find container \"94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411\": container with ID starting with 94d3ef16b17c8ca4f7ad5006fb8d1b587e7727ba3977a2ff8b6545de96eb4411 not found: ID does not exist"
Oct 30 22:49:14.175508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.175476    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"]
Oct 30 22:49:14.184725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:14.184696    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-86ztv"]
Oct 30 22:49:15.285231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.285190    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:15.326115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.326084    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-openshift-ca\") pod \"f79fd7df-a950-4f68-868c-e8af827e830d\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") "
Oct 30 22:49:15.326300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.326176    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79fd7df-a950-4f68-868c-e8af827e830d-kube-api-access-wpxk7\" (UniqueName: \"kubernetes.io/projected/f79fd7df-a950-4f68-868c-e8af827e830d-kube-api-access-wpxk7\") pod \"f79fd7df-a950-4f68-868c-e8af827e830d\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") "
Oct 30 22:49:15.326300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.326293    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-xks-ca\") pod \"f79fd7df-a950-4f68-868c-e8af827e830d\" (UID: \"f79fd7df-a950-4f68-868c-e8af827e830d\") "
Oct 30 22:49:15.326578 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.326551    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-openshift-ca" pod "f79fd7df-a950-4f68-868c-e8af827e830d" (UID: "f79fd7df-a950-4f68-868c-e8af827e830d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:15.326875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.326854    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:15.328584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.328558    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79fd7df-a950-4f68-868c-e8af827e830d-kube-api-access-wpxk7" pod "f79fd7df-a950-4f68-868c-e8af827e830d" (UID: "f79fd7df-a950-4f68-868c-e8af827e830d"). InnerVolumeSpecName "kube-api-access-wpxk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:15.328694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.328595    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-xks-ca" pod "f79fd7df-a950-4f68-868c-e8af827e830d" (UID: "f79fd7df-a950-4f68-868c-e8af827e830d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:15.428038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.427958    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79fd7df-a950-4f68-868c-e8af827e830d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:15.428038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:15.427991    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxk7\" (UniqueName: \"kubernetes.io/projected/f79fd7df-a950-4f68-868c-e8af827e830d-kube-api-access-wpxk7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:16.099065 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:16.099030    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" path="/var/lib/kubelet/pods/f7a61397-afcd-4557-80c6-de16c4023d99/volumes"
Oct 30 22:49:16.120656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:16.120630    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"
Oct 30 22:49:16.120798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:16.120657    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl" event={"ID":"f79fd7df-a950-4f68-868c-e8af827e830d","Type":"ContainerDied","Data":"f1f69da369f472a4c89e8142f5b714ecaeba39d38ce7dfc249584bdde14edfd5"}
Oct 30 22:49:16.120798 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:16.120711    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1f69da369f472a4c89e8142f5b714ecaeba39d38ce7dfc249584bdde14edfd5"
Oct 30 22:49:22.857431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:22.857390    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"]
Oct 30 22:49:24.484294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:24.484248    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"]
Oct 30 22:49:24.488139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:24.488107    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bc48cc-z5cgl"]
Oct 30 22:49:26.100153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:26.100119    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79fd7df-a950-4f68-868c-e8af827e830d" path="/var/lib/kubelet/pods/f79fd7df-a950-4f68-868c-e8af827e830d/volumes"
Oct 30 22:49:27.858796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:27.858727    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7" podUID="71ab1e84-d213-4926-acf5-1376f4cfc2fd" containerName="payload-pre-processing" containerID="cri-o://8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525" gracePeriod=25
Oct 30 22:49:31.593442 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:31.593398    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"]
Oct 30 22:49:31.593969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:31.593670    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy" containerID="cri-o://3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0" gracePeriod=30
Oct 30 22:49:31.997574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:31.997533    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7" podUID="71ab1e84-d213-4926-acf5-1376f4cfc2fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:32.156787 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.156753    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"]
Oct 30 22:49:32.157389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157376    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy"
Oct 30 22:49:32.157451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157391    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy"
Oct 30 22:49:32.157451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157420    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f79fd7df-a950-4f68-868c-e8af827e830d" containerName="revoke-keys"
Oct 30 22:49:32.157451 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157425    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79fd7df-a950-4f68-868c-e8af827e830d" containerName="revoke-keys"
Oct 30 22:49:32.157557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157459    4581 cpu_manager.go:414] "containerMap: removing container" podUID="818ad606-0326-4722-b613-f39cbbdc392a" containerName="revoke-keys"
Oct 30 22:49:32.157557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157465    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818ad606-0326-4722-b613-f39cbbdc392a" containerName="revoke-keys"
Oct 30 22:49:32.157667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157581    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy"
Oct 30 22:49:32.157667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157592    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="818ad606-0326-4722-b613-f39cbbdc392a" containerName="revoke-keys"
Oct 30 22:49:32.157667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157602    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy"
Oct 30 22:49:32.157667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.157612    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79fd7df-a950-4f68-868c-e8af827e830d" containerName="revoke-keys"
Oct 30 22:49:32.161206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.161186    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.175606 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.175585    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:32.175774 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.175658    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:32.181535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.181505    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"]
Oct 30 22:49:32.215438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.215401    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:32.215619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.215465    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:49:32.344175 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.344093    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5df\" (UniqueName: \"kubernetes.io/projected/bd8f2736-faef-457b-b9bb-ef0953baff56-kube-api-access-sn5df\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.344369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.344303    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.344421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.344369    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.445506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.445473    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5df\" (UniqueName: \"kubernetes.io/projected/bd8f2736-faef-457b-b9bb-ef0953baff56-kube-api-access-sn5df\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.445679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.445578    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.445679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.445628    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.446208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.446184    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.448146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.448124    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.462644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.462604    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5df\" (UniqueName: \"kubernetes.io/projected/bd8f2736-faef-457b-b9bb-ef0953baff56-kube-api-access-sn5df\") pod \"maas-api-revoke-keys-e2e-inf-b-867f12-cckkg\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.477570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.477320    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:32.822319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:32.822282    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"]
Oct 30 22:49:32.830544 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:49:32.830506    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8f2736_faef_457b_b9bb_ef0953baff56.slice/crio-d75a1a7eefcc8c5973b10c17ce076316819fdec9e601886b791c14d57ef0eec3 WatchSource:0}: Error finding container d75a1a7eefcc8c5973b10c17ce076316819fdec9e601886b791c14d57ef0eec3: Status 404 returned error can't find the container with id d75a1a7eefcc8c5973b10c17ce076316819fdec9e601886b791c14d57ef0eec3
Oct 30 22:49:33.324258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:33.323773    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg" event={"ID":"bd8f2736-faef-457b-b9bb-ef0953baff56","Type":"ContainerStarted","Data":"cf04d98797edc7ef2ee4f10634c3a89b194ae3989f924708ae94cafd7ffa9e41"}
Oct 30 22:49:33.324258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:33.323841    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg" event={"ID":"bd8f2736-faef-457b-b9bb-ef0953baff56","Type":"ContainerStarted","Data":"d75a1a7eefcc8c5973b10c17ce076316819fdec9e601886b791c14d57ef0eec3"}
Oct 30 22:49:33.363199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:33.363132    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg" podStartSLOduration=1.363116939 podStartE2EDuration="1.363116939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:33.359324028 +0000 UTC m=+2557.398262917" watchObservedRunningTime="2026-10-30 22:49:33.363116939 +0000 UTC m=+2557.402055825"
Oct 30 22:49:34.371244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:34.366298    4581 generic.go:356] "Generic (PLEG): container finished" podID="bd8f2736-faef-457b-b9bb-ef0953baff56" containerID="cf04d98797edc7ef2ee4f10634c3a89b194ae3989f924708ae94cafd7ffa9e41" exitCode=0
Oct 30 22:49:34.371244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:34.366543    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg" event={"ID":"bd8f2736-faef-457b-b9bb-ef0953baff56","Type":"ContainerDied","Data":"cf04d98797edc7ef2ee4f10634c3a89b194ae3989f924708ae94cafd7ffa9e41"}
Oct 30 22:49:35.689678 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.689608    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:35.736960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.736920    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-openshift-ca\") pod \"bd8f2736-faef-457b-b9bb-ef0953baff56\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") "
Oct 30 22:49:35.737150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.737136    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-xks-ca\") pod \"bd8f2736-faef-457b-b9bb-ef0953baff56\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") "
Oct 30 22:49:35.737198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.737178    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8f2736-faef-457b-b9bb-ef0953baff56-kube-api-access-sn5df\" (UniqueName: \"kubernetes.io/projected/bd8f2736-faef-457b-b9bb-ef0953baff56-kube-api-access-sn5df\") pod \"bd8f2736-faef-457b-b9bb-ef0953baff56\" (UID: \"bd8f2736-faef-457b-b9bb-ef0953baff56\") "
Oct 30 22:49:35.737979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.737950    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-openshift-ca" pod "bd8f2736-faef-457b-b9bb-ef0953baff56" (UID: "bd8f2736-faef-457b-b9bb-ef0953baff56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:35.741271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.741186    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-xks-ca" pod "bd8f2736-faef-457b-b9bb-ef0953baff56" (UID: "bd8f2736-faef-457b-b9bb-ef0953baff56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:35.741459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.741416    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8f2736-faef-457b-b9bb-ef0953baff56-kube-api-access-sn5df" pod "bd8f2736-faef-457b-b9bb-ef0953baff56" (UID: "bd8f2736-faef-457b-b9bb-ef0953baff56"). InnerVolumeSpecName "kube-api-access-sn5df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:35.839009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.838973    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:35.839009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.839006    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn5df\" (UniqueName: \"kubernetes.io/projected/bd8f2736-faef-457b-b9bb-ef0953baff56-kube-api-access-sn5df\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:35.839009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:35.839016    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f2736-faef-457b-b9bb-ef0953baff56-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:36.405271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:36.405168    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg" event={"ID":"bd8f2736-faef-457b-b9bb-ef0953baff56","Type":"ContainerDied","Data":"d75a1a7eefcc8c5973b10c17ce076316819fdec9e601886b791c14d57ef0eec3"}
Oct 30 22:49:36.405271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:36.405205    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"
Oct 30 22:49:36.405271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:36.405234    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75a1a7eefcc8c5973b10c17ce076316819fdec9e601886b791c14d57ef0eec3"
Oct 30 22:49:36.998311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:36.998186    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195023    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istiod-ca-cert\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195130    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-token\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-token\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195195    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-envoy\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195301    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-kube-api-access-ft954\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-kube-api-access-ft954\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195339    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-credential-socket\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195424    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-data\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195502    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-certs\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195563    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-podinfo\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195634    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-socket\") pod \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\" (UID: \"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f\") "
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195660    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istiod-ca-cert" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:37.196392 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.195996    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-credential-socket" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.197269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.196601    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.197269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.196632    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.197269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.196858    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-socket" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.197269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.197039    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-data" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.197269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.197236    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-certs" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.209001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.208680    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-token" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:37.210364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.210183    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-envoy" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:37.210733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.210708    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-podinfo" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:37.213286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.213258    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-kube-api-access-ft954" pod "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" (UID: "8c722c1e-996e-4e92-9d8b-0b100e7c8d3f"). InnerVolumeSpecName "kube-api-access-ft954". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:37.298158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.298069    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.298158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.298101    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.298158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.298111    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft954\" (UniqueName: \"kubernetes.io/projected/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-kube-api-access-ft954\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.298158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.298120    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.298158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.298129    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.298158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.298137    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.298158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.298145    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:37.420615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.420578    4581 generic.go:356] "Generic (PLEG): container finished" podID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerID="3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0" exitCode=0
Oct 30 22:49:37.420927 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.420637    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"
Oct 30 22:49:37.421035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.420654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" event={"ID":"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f","Type":"ContainerDied","Data":"3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0"}
Oct 30 22:49:37.421201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.421184    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw" event={"ID":"8c722c1e-996e-4e92-9d8b-0b100e7c8d3f","Type":"ContainerDied","Data":"61660af84f1e67c1bd631095162eeb88e7f04bdc621a6c3109cf2397ed568a6e"}
Oct 30 22:49:37.421387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.421358    4581 scope.go:122] "RemoveContainer" containerID="3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0"
Oct 30 22:49:37.438555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.438534    4581 scope.go:122] "RemoveContainer" containerID="80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d"
Oct 30 22:49:37.450054 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.450022    4581 scope.go:122] "RemoveContainer" containerID="3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0"
Oct 30 22:49:37.480850 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:37.480383    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0\": container with ID starting with 3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0 not found: ID does not exist" containerID="3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0"
Oct 30 22:49:37.480850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.480445    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0"} err="failed to get container status \"3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0\": rpc error: code = NotFound desc = could not find container \"3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0\": container with ID starting with 3d0c220d394b06b9cc724ab3b9c5a276da9a49d9d14b4db8234a53bfd47d2bd0 not found: ID does not exist"
Oct 30 22:49:37.480850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.480475    4581 scope.go:122] "RemoveContainer" containerID="80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d"
Oct 30 22:49:37.486243 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:37.482377    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d\": container with ID starting with 80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d not found: ID does not exist" containerID="80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d"
Oct 30 22:49:37.486243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.482421    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d"} err="failed to get container status \"80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d\": rpc error: code = NotFound desc = could not find container \"80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d\": container with ID starting with 80276fa38cc4175b4cdc29979b310ff8507a975573ea9e5e920a4e4685595d8d not found: ID does not exist"
Oct 30 22:49:37.486243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.485862    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"]
Oct 30 22:49:37.490788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:37.490764    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bc48cc-openshift-default-d9498d65-kbdtw"]
Oct 30 22:49:38.099297 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:38.099260    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" path="/var/lib/kubelet/pods/8c722c1e-996e-4e92-9d8b-0b100e7c8d3f/volumes"
Oct 30 22:49:40.820237 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:40.818829    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:49:42.014246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:42.009514    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7" podUID="71ab1e84-d213-4926-acf5-1376f4cfc2fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:42.610850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:42.610789    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"]
Oct 30 22:49:44.511761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:44.511726    4581 generic.go:356] "Generic (PLEG): container finished" podID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerID="4bf414d2c5cc881ac7a528e9e4fa85158c7bc366f69a53ebe6066e5c5a669d49" exitCode=0
Oct 30 22:49:44.512351 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:44.511802    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" event={"ID":"f91815b9-ea0e-4d30-b6fb-72e9933362cf","Type":"ContainerDied","Data":"4bf414d2c5cc881ac7a528e9e4fa85158c7bc366f69a53ebe6066e5c5a669d49"}
Oct 30 22:49:44.512435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:44.512387    4581 scope.go:122] "RemoveContainer" containerID="4bf414d2c5cc881ac7a528e9e4fa85158c7bc366f69a53ebe6066e5c5a669d49"
Oct 30 22:49:44.514150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:44.514122    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:44.514249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:44.514193    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:44.514249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:44.514243    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:45.638558 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:45.638012    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" event={"ID":"f91815b9-ea0e-4d30-b6fb-72e9933362cf","Type":"ContainerStarted","Data":"8c82898fbeee21507fcef8fff68b17d10099b7bdb5772551e282034624391627"}
Oct 30 22:49:46.107926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.107894    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"]
Oct 30 22:49:46.108097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.107939    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3acfdb-7zmsq"]
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.195884    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"]
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.196828    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.196851    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a61397-afcd-4557-80c6-de16c4023d99" containerName="istio-proxy"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.196876    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bd8f2736-faef-457b-b9bb-ef0953baff56" containerName="revoke-keys"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.196885    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8f2736-faef-457b-b9bb-ef0953baff56" containerName="revoke-keys"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.196943    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.196951    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.196967    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.196975    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.197144    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8f2736-faef-457b-b9bb-ef0953baff56" containerName="revoke-keys"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.197174    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy"
Oct 30 22:49:46.197608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.197185    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c722c1e-996e-4e92-9d8b-0b100e7c8d3f" containerName="istio-proxy"
Oct 30 22:49:46.201552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.201525    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.231929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.231662    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-23aab2-openshift-default-dockercfg-4ltc5\""
Oct 30 22:49:46.232767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.232745    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"]
Oct 30 22:49:46.295303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.294111    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-socket\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.295477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.295303    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2754d3d-bedb-443c-95a2-db9070f4ed02-istiod-ca-cert\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.295477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.295403    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-certs\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.295561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.295489    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-token\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.295561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.295535    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-data\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.295632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.295598    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svqk2\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-kube-api-access-svqk2\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.295668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.295634    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-podinfo\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.295752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.295730    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-envoy\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.295958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.295940    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-credential-socket\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.421291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.420733    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-certs\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.420291    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-certs\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.430404    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-token\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.430454    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-data\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.430528    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svqk2\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-kube-api-access-svqk2\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.430570    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-podinfo\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.430673    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-envoy\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.430742    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-credential-socket\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.430820    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-socket\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.430850    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2754d3d-bedb-443c-95a2-db9070f4ed02-istiod-ca-cert\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.431762    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2754d3d-bedb-443c-95a2-db9070f4ed02-istiod-ca-cert\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.434244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.432413    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-data\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.452889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.452855    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-envoy\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.453417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.453392    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-credential-socket\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.453837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.453813    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-socket\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.467042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.467013    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-podinfo\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.508269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.507942    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svqk2\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-kube-api-access-svqk2\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.541272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.539748    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:49:46.543002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.541533    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 22:49:46.543002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.541586    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 22:49:46.606250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.603836    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-token\") pod \"e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") " pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.831290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.831160    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:46.995246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:46.994368    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:49:47.209170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.209136    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"]
Oct 30 22:49:47.209871 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:49:47.209845    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2754d3d_bedb_443c_95a2_db9070f4ed02.slice/crio-b527b46911f1df4c2dc5af03f373296a9e248731aced5d74761531c1c61ccf50 WatchSource:0}: Error finding container b527b46911f1df4c2dc5af03f373296a9e248731aced5d74761531c1c61ccf50: Status 404 returned error can't find the container with id b527b46911f1df4c2dc5af03f373296a9e248731aced5d74761531c1c61ccf50
Oct 30 22:49:47.213016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.212581    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:47.213016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.212653    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:47.213016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.212685    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:49:47.532817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.532695    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 22:49:47.532817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.532769    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 22:49:47.611936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.611893    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q" podUID="ca001362-beab-4a6e-858b-bd735928f157" containerName="payload-pre-processing" containerID="cri-o://868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4" gracePeriod=25
Oct 30 22:49:47.684695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.684658    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" event={"ID":"e2754d3d-bedb-443c-95a2-db9070f4ed02","Type":"ContainerStarted","Data":"c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b"}
Oct 30 22:49:47.685102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.685081    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" event={"ID":"e2754d3d-bedb-443c-95a2-db9070f4ed02","Type":"ContainerStarted","Data":"b527b46911f1df4c2dc5af03f373296a9e248731aced5d74761531c1c61ccf50"}
Oct 30 22:49:47.735205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.735141    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podStartSLOduration=1.735118787 podStartE2EDuration="1.735118787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:47.732211482 +0000 UTC m=+2571.771150367" watchObservedRunningTime="2026-10-30 22:49:47.735118787 +0000 UTC m=+2571.774057673"
Oct 30 22:49:47.831302 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:47.831210    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:48.104966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.104932    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818ad606-0326-4722-b613-f39cbbdc392a" path="/var/lib/kubelet/pods/818ad606-0326-4722-b613-f39cbbdc392a/volumes"
Oct 30 22:49:48.238240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.237646    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:49:48.331557 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.331508    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q" podUID="ca001362-beab-4a6e-858b-bd735928f157" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:48.411851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.411818    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ab1e84-d213-4926-acf5-1376f4cfc2fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab1e84-d213-4926-acf5-1376f4cfc2fd-plugins-config-volume\") pod \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\" (UID: \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\") "
Oct 30 22:49:48.412058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.412009    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ab1e84-d213-4926-acf5-1376f4cfc2fd-kube-api-access-q6nlf\" (UniqueName: \"kubernetes.io/projected/71ab1e84-d213-4926-acf5-1376f4cfc2fd-kube-api-access-q6nlf\") pod \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\" (UID: \"71ab1e84-d213-4926-acf5-1376f4cfc2fd\") "
Oct 30 22:49:48.412182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.412157    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ab1e84-d213-4926-acf5-1376f4cfc2fd-plugins-config-volume" pod "71ab1e84-d213-4926-acf5-1376f4cfc2fd" (UID: "71ab1e84-d213-4926-acf5-1376f4cfc2fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:48.412542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.412523    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ab1e84-d213-4926-acf5-1376f4cfc2fd-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:48.414433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.414410    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ab1e84-d213-4926-acf5-1376f4cfc2fd-kube-api-access-q6nlf" pod "71ab1e84-d213-4926-acf5-1376f4cfc2fd" (UID: "71ab1e84-d213-4926-acf5-1376f4cfc2fd"). InnerVolumeSpecName "kube-api-access-q6nlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:48.513663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.513627    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6nlf\" (UniqueName: \"kubernetes.io/projected/71ab1e84-d213-4926-acf5-1376f4cfc2fd-kube-api-access-q6nlf\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:48.532596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.532558    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 22:49:48.532783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.532607    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 22:49:48.704312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.703717    4581 generic.go:356] "Generic (PLEG): container finished" podID="71ab1e84-d213-4926-acf5-1376f4cfc2fd" containerID="8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525" exitCode=0
Oct 30 22:49:48.704312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.703917    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7" event={"ID":"71ab1e84-d213-4926-acf5-1376f4cfc2fd","Type":"ContainerDied","Data":"8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525"}
Oct 30 22:49:48.704312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.703945    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"
Oct 30 22:49:48.704312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.704025    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7" event={"ID":"71ab1e84-d213-4926-acf5-1376f4cfc2fd","Type":"ContainerDied","Data":"fa54f7c90048d36cbe4a7a5ae049d27cd5a457e0fa65fc577d6eb13f1d8f49a8"}
Oct 30 22:49:48.704312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.704071    4581 scope.go:122] "RemoveContainer" containerID="8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525"
Oct 30 22:49:48.719999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.719979    4581 scope.go:122] "RemoveContainer" containerID="8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525"
Oct 30 22:49:48.720447 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:48.720414    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525\": container with ID starting with 8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525 not found: ID does not exist" containerID="8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525"
Oct 30 22:49:48.720584 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.720457    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525"} err="failed to get container status \"8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525\": rpc error: code = NotFound desc = could not find container \"8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525\": container with ID starting with 8616e70b0bb43b48cbd6aa9ade4cb6b88eebbdc70c2c945386a4639e15e6e525 not found: ID does not exist"
Oct 30 22:49:48.788602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.788548    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"]
Oct 30 22:49:48.794454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.794425    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bc48cc-5554659f79-79xg7"]
Oct 30 22:49:48.837537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:48.837510    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:49.532876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:49.532841    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 22:49:49.533064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:49.532919    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 22:49:49.715537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:49.715510    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:49.716445 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:49.716423    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:49:50.099227 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:50.099170    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ab1e84-d213-4926-acf5-1376f4cfc2fd" path="/var/lib/kubelet/pods/71ab1e84-d213-4926-acf5-1376f4cfc2fd/volumes"
Oct 30 22:49:50.532934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:50.532881    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 22:49:50.533153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:50.532950    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 22:49:51.537268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:51.537241    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:49:51.538565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:51.538541    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 22:49:52.034487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.034455    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"]
Oct 30 22:49:52.035062 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.035050    4581 cpu_manager.go:414] "containerMap: removing container" podUID="71ab1e84-d213-4926-acf5-1376f4cfc2fd" containerName="payload-pre-processing"
Oct 30 22:49:52.035105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.035065    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ab1e84-d213-4926-acf5-1376f4cfc2fd" containerName="payload-pre-processing"
Oct 30 22:49:52.035256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.035207    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ab1e84-d213-4926-acf5-1376f4cfc2fd" containerName="payload-pre-processing"
Oct 30 22:49:52.038740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.038718    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:52.053060 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.053033    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-23aab2-dockercfg-9r494\""
Oct 30 22:49:52.053370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.053355    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-23aab2\""
Oct 30 22:49:52.059934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.059909    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"]
Oct 30 22:49:52.064159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.064135    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd4230d-82fc-4310-a897-744c6deb6404-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r\" (UID: \"9cd4230d-82fc-4310-a897-744c6deb6404\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:52.064330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.064311    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s4vx\" (UniqueName: \"kubernetes.io/projected/9cd4230d-82fc-4310-a897-744c6deb6404-kube-api-access-2s4vx\") pod \"payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r\" (UID: \"9cd4230d-82fc-4310-a897-744c6deb6404\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:52.248081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.194639    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd4230d-82fc-4310-a897-744c6deb6404-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r\" (UID: \"9cd4230d-82fc-4310-a897-744c6deb6404\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:52.248682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.248649    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4vx\" (UniqueName: \"kubernetes.io/projected/9cd4230d-82fc-4310-a897-744c6deb6404-kube-api-access-2s4vx\") pod \"payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r\" (UID: \"9cd4230d-82fc-4310-a897-744c6deb6404\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:52.249479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.207105    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd4230d-82fc-4310-a897-744c6deb6404-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r\" (UID: \"9cd4230d-82fc-4310-a897-744c6deb6404\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:52.328983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.328889    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"]
Oct 30 22:49:52.342588 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.342563    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"]
Oct 30 22:49:52.343056 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.342927    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4vx\" (UniqueName: \"kubernetes.io/projected/9cd4230d-82fc-4310-a897-744c6deb6404-kube-api-access-2s4vx\") pod \"payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r\" (UID: \"9cd4230d-82fc-4310-a897-744c6deb6404\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:52.343169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.343077    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:52.359259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.357599    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:52.454240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.454177    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dqk5\" (UniqueName: \"kubernetes.io/projected/407ec7cd-8278-4a86-b8ff-0fb8500b531a-kube-api-access-6dqk5\") pod \"payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29\" (UID: \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:52.454400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.454343    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407ec7cd-8278-4a86-b8ff-0fb8500b531a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29\" (UID: \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:52.556203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.556162    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dqk5\" (UniqueName: \"kubernetes.io/projected/407ec7cd-8278-4a86-b8ff-0fb8500b531a-kube-api-access-6dqk5\") pod \"payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29\" (UID: \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:52.556711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.556275    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407ec7cd-8278-4a86-b8ff-0fb8500b531a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29\" (UID: \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:52.557182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.557155    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407ec7cd-8278-4a86-b8ff-0fb8500b531a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29\" (UID: \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:52.574781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.574749    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dqk5\" (UniqueName: \"kubernetes.io/projected/407ec7cd-8278-4a86-b8ff-0fb8500b531a-kube-api-access-6dqk5\") pod \"payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29\" (UID: \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:52.680129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.680093    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"]
Oct 30 22:49:52.684658 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:49:52.684626    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd4230d_82fc_4310_a897_744c6deb6404.slice/crio-648a5dee217c07065f44cd2b893e01d0f446a383bf4019f28faee3878c4e9fe4 WatchSource:0}: Error finding container 648a5dee217c07065f44cd2b893e01d0f446a383bf4019f28faee3878c4e9fe4: Status 404 returned error can't find the container with id 648a5dee217c07065f44cd2b893e01d0f446a383bf4019f28faee3878c4e9fe4
Oct 30 22:49:52.710658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.710630    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:52.755277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:52.755241    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r" event={"ID":"9cd4230d-82fc-4310-a897-744c6deb6404","Type":"ContainerStarted","Data":"648a5dee217c07065f44cd2b893e01d0f446a383bf4019f28faee3878c4e9fe4"}
Oct 30 22:49:53.026966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:53.026825    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"]
Oct 30 22:49:53.038229 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:49:53.038187    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407ec7cd_8278_4a86_b8ff_0fb8500b531a.slice/crio-df085f9708885646d9a2767a1c5ac627833ec656a6a4edbc7b41a6910b1fad1a WatchSource:0}: Error finding container df085f9708885646d9a2767a1c5ac627833ec656a6a4edbc7b41a6910b1fad1a: Status 404 returned error can't find the container with id df085f9708885646d9a2767a1c5ac627833ec656a6a4edbc7b41a6910b1fad1a
Oct 30 22:49:53.768172 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:53.768131    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29" event={"ID":"407ec7cd-8278-4a86-b8ff-0fb8500b531a","Type":"ContainerStarted","Data":"1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097"}
Oct 30 22:49:53.768708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:53.768201    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29" event={"ID":"407ec7cd-8278-4a86-b8ff-0fb8500b531a","Type":"ContainerStarted","Data":"df085f9708885646d9a2767a1c5ac627833ec656a6a4edbc7b41a6910b1fad1a"}
Oct 30 22:49:53.768708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:53.768263    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:49:53.770278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:53.770251    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r" event={"ID":"9cd4230d-82fc-4310-a897-744c6deb6404","Type":"ContainerStarted","Data":"a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0"}
Oct 30 22:49:53.770437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:53.770310    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:49:53.838679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:53.831592    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29" podStartSLOduration=1.831570448 podStartE2EDuration="1.831570448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:53.813744721 +0000 UTC m=+2577.852683608" watchObservedRunningTime="2026-10-30 22:49:53.831570448 +0000 UTC m=+2577.870509335"
Oct 30 22:49:53.856740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:53.856674    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r" podStartSLOduration=1.8566606110000001 podStartE2EDuration="1.856660611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:53.853449219 +0000 UTC m=+2577.892388116" watchObservedRunningTime="2026-10-30 22:49:53.856660611 +0000 UTC m=+2577.895599497"
Oct 30 22:49:54.305920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:54.305882    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"]
Oct 30 22:49:54.306660 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:54.306208    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy" containerID="cri-o://f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c" gracePeriod=30
Oct 30 22:49:58.331987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:58.331948    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q" podUID="ca001362-beab-4a6e-858b-bd735928f157" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:59.590759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.590693    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:49:59.688210 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688172    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-credential-socket\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688355    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istiod-ca-cert\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688436    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-certs\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688475    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-token\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-token\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688502    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-credential-socket" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.688602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688527    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-podinfo\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688565    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-socket\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688647    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-data\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688693    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-kube-api-access-6976r\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-kube-api-access-6976r\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688722    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-envoy\") pod \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\" (UID: \"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2\") "
Oct 30 22:49:59.688760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.688740    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-certs" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.689064 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.689042    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istiod-ca-cert" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:59.689103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.689043    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-socket" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.689369 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.689335    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-data" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.690347 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.690320    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.690497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.690351    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.690497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.690365    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.690497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.690378    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.690497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.690391    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.692055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.692016    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-podinfo" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:59.692173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.692142    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-kube-api-access-6976r" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "kube-api-access-6976r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:59.692752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.692727    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-token" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:59.692852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.692805    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-envoy" pod "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" (UID: "0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:59.791701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.791666    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.791701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.791698    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.791701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.791710    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6976r\" (UniqueName: \"kubernetes.io/projected/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-kube-api-access-6976r\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.791948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.791722    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:49:59.831563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.831529    4581 generic.go:356] "Generic (PLEG): container finished" podID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerID="f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c" exitCode=0
Oct 30 22:49:59.831736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.831600    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" event={"ID":"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2","Type":"ContainerDied","Data":"f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c"}
Oct 30 22:49:59.831736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.831613    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"
Oct 30 22:49:59.831736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.831705    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm" event={"ID":"0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2","Type":"ContainerDied","Data":"740ef2eb6fe4f5e3ac0907317d49a69307ecf09b6d3ef846361ea937e96b5279"}
Oct 30 22:49:59.831736 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.831735    4581 scope.go:122] "RemoveContainer" containerID="f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c"
Oct 30 22:49:59.845291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.845270    4581 scope.go:122] "RemoveContainer" containerID="7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f"
Oct 30 22:49:59.856468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.856449    4581 scope.go:122] "RemoveContainer" containerID="f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c"
Oct 30 22:49:59.856793 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:59.856770    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c\": container with ID starting with f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c not found: ID does not exist" containerID="f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c"
Oct 30 22:49:59.856905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.856798    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c"} err="failed to get container status \"f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c\": rpc error: code = NotFound desc = could not find container \"f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c\": container with ID starting with f093efc11cfd6f5c22dbb10ca53a33595a6131e8fcf4287c36e15608533ab15c not found: ID does not exist"
Oct 30 22:49:59.856905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.856816    4581 scope.go:122] "RemoveContainer" containerID="7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f"
Oct 30 22:49:59.857074 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:49:59.857050    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f\": container with ID starting with 7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f not found: ID does not exist" containerID="7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f"
Oct 30 22:49:59.857131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.857081    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f"} err="failed to get container status \"7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f\": rpc error: code = NotFound desc = could not find container \"7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f\": container with ID starting with 7d8242e986ce84ccbe0fd8b816d1fc02c4642eb02a8955589d167dec850cc79f not found: ID does not exist"
Oct 30 22:49:59.881352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.881327    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"]
Oct 30 22:49:59.887435 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:49:59.887402    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3acfdb-openshift-default-5b477b885f-mrqbm"]
Oct 30 22:50:00.099325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:00.099240    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" path="/var/lib/kubelet/pods/0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2/volumes"
Oct 30 22:50:01.053909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:01.053875    4581 scope.go:122] "RemoveContainer" containerID="8d31ac090cf826ef6985071d4148a59ebba8b3e72cba5f0c286d7714da9f9a6a"
Oct 30 22:50:01.065703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:01.065680    4581 scope.go:122] "RemoveContainer" containerID="7110d24ca1c4e1f3d62d4852aefaf743c32e034a72b63bb8c39a711b1f140cf1"
Oct 30 22:50:04.599971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:04.599935    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"]
Oct 30 22:50:04.608465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:04.608438    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-867f12-cckkg"]
Oct 30 22:50:04.782569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:04.782542    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 22:50:04.782752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:04.782616    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 22:50:06.099777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:06.099743    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8f2736-faef-457b-b9bb-ef0953baff56" path="/var/lib/kubelet/pods/bd8f2736-faef-457b-b9bb-ef0953baff56/volumes"
Oct 30 22:50:07.902836 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.902803    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:50:07.924814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.924779    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca001362-beab-4a6e-858b-bd735928f157-kube-api-access-vhj7v\" (UniqueName: \"kubernetes.io/projected/ca001362-beab-4a6e-858b-bd735928f157-kube-api-access-vhj7v\") pod \"ca001362-beab-4a6e-858b-bd735928f157\" (UID: \"ca001362-beab-4a6e-858b-bd735928f157\") "
Oct 30 22:50:07.925009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.924947    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca001362-beab-4a6e-858b-bd735928f157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca001362-beab-4a6e-858b-bd735928f157-plugins-config-volume\") pod \"ca001362-beab-4a6e-858b-bd735928f157\" (UID: \"ca001362-beab-4a6e-858b-bd735928f157\") "
Oct 30 22:50:07.925347 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.925305    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca001362-beab-4a6e-858b-bd735928f157-plugins-config-volume" pod "ca001362-beab-4a6e-858b-bd735928f157" (UID: "ca001362-beab-4a6e-858b-bd735928f157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:07.925781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.925752    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca001362-beab-4a6e-858b-bd735928f157-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:07.927764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.927731    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca001362-beab-4a6e-858b-bd735928f157-kube-api-access-vhj7v" pod "ca001362-beab-4a6e-858b-bd735928f157" (UID: "ca001362-beab-4a6e-858b-bd735928f157"). InnerVolumeSpecName "kube-api-access-vhj7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:07.947769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.947734    4581 generic.go:356] "Generic (PLEG): container finished" podID="ca001362-beab-4a6e-858b-bd735928f157" containerID="868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4" exitCode=0
Oct 30 22:50:07.947924 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.947805    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"
Oct 30 22:50:07.948096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.948070    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q" event={"ID":"ca001362-beab-4a6e-858b-bd735928f157","Type":"ContainerDied","Data":"868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4"}
Oct 30 22:50:07.948179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.948131    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q" event={"ID":"ca001362-beab-4a6e-858b-bd735928f157","Type":"ContainerDied","Data":"1e5fbaed011eb3d927cd51bc7a2f31b33daa1c8d06eda17401d14d3fd1030821"}
Oct 30 22:50:07.948179 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.948161    4581 scope.go:122] "RemoveContainer" containerID="868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4"
Oct 30 22:50:07.968644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.968619    4581 scope.go:122] "RemoveContainer" containerID="868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4"
Oct 30 22:50:07.968932 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:50:07.968913    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4\": container with ID starting with 868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4 not found: ID does not exist" containerID="868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4"
Oct 30 22:50:07.969011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.968939    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4"} err="failed to get container status \"868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4\": rpc error: code = NotFound desc = could not find container \"868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4\": container with ID starting with 868c43a73b07b73304407e0744e36677814fe9243428d837c80fc7eb396fa6f4 not found: ID does not exist"
Oct 30 22:50:07.998097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:07.998055    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"]
Oct 30 22:50:08.008729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:08.008682    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-867f12-f59b9bd47-xkz7q"]
Oct 30 22:50:08.027304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:08.027266    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhj7v\" (UniqueName: \"kubernetes.io/projected/ca001362-beab-4a6e-858b-bd735928f157-kube-api-access-vhj7v\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:08.101130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:08.101093    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca001362-beab-4a6e-858b-bd735928f157" path="/var/lib/kubelet/pods/ca001362-beab-4a6e-858b-bd735928f157/volumes"
Oct 30 22:50:10.820335 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:50:10.820292    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:50:11.980999 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:11.980962    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"]
Oct 30 22:50:11.981488 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:11.981319    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy" containerID="cri-o://7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6" gracePeriod=30
Oct 30 22:50:13.210824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.210790    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"]
Oct 30 22:50:13.211471 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.211452    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy"
Oct 30 22:50:13.211556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.211472    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy"
Oct 30 22:50:13.211556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.211484    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ca001362-beab-4a6e-858b-bd735928f157" containerName="payload-pre-processing"
Oct 30 22:50:13.211556 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.211489    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca001362-beab-4a6e-858b-bd735928f157" containerName="payload-pre-processing"
Oct 30 22:50:13.211723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.211640    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy"
Oct 30 22:50:13.211723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.211649    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca001362-beab-4a6e-858b-bd735928f157" containerName="payload-pre-processing"
Oct 30 22:50:13.211723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.211657    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy"
Oct 30 22:50:13.216657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.216635    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.230086 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.230066    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cdc122-openshift-default-dockercfg-4sxwk\""
Oct 30 22:50:13.262283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.262248    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"]
Oct 30 22:50:13.316694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.316658    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.316694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.316696    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.316915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.316835    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.316915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.316891    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edd8469-95e9-41ee-a592-9c6c20aaff03-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.317006 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.316986    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.317085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.317069    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.317122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.317109    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.317160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.317128    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.317160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.317152    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvl8n\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-kube-api-access-lvl8n\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418347 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418309    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edd8469-95e9-41ee-a592-9c6c20aaff03-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418360    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418527 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418477    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418539    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418562    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418596    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvl8n\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-kube-api-access-lvl8n\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418671    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418733 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418703    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418761    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418825    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.418944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.418887    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.419115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.419092    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edd8469-95e9-41ee-a592-9c6c20aaff03-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.419269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.419251    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.419501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.419483    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.421076 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.421051    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.421417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.421400    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.437146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.437121    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.437277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.437189    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvl8n\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-kube-api-access-lvl8n\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.541321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.540993    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:13.935384 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.935356    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"]
Oct 30 22:50:13.937644 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:50:13.937591    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edd8469_95e9_41ee_a592_9c6c20aaff03.slice/crio-1bf93d2ea945a286f2d72c334052aae05688e3e2e97de83d934e05d38c8582d7 WatchSource:0}: Error finding container 1bf93d2ea945a286f2d72c334052aae05688e3e2e97de83d934e05d38c8582d7: Status 404 returned error can't find the container with id 1bf93d2ea945a286f2d72c334052aae05688e3e2e97de83d934e05d38c8582d7
Oct 30 22:50:13.940904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.940865    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:50:13.941023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.940950    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:50:13.941023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:13.940993    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:50:14.020245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:14.020201    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" event={"ID":"3edd8469-95e9-41ee-a592-9c6c20aaff03","Type":"ContainerStarted","Data":"1bf93d2ea945a286f2d72c334052aae05688e3e2e97de83d934e05d38c8582d7"}
Oct 30 22:50:15.029726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:15.029674    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" event={"ID":"3edd8469-95e9-41ee-a592-9c6c20aaff03","Type":"ContainerStarted","Data":"3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d"}
Oct 30 22:50:15.088965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:15.088904    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" podStartSLOduration=2.088885903 podStartE2EDuration="2.088885903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:15.075730822 +0000 UTC m=+2599.114669718" watchObservedRunningTime="2026-10-30 22:50:15.088885903 +0000 UTC m=+2599.127824790"
Oct 30 22:50:15.541454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:15.541412    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:15.547207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:15.547173    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:16.037855 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:16.037823    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:16.038895 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:16.038870    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:50:17.262298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.262229    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:50:17.390434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.390402    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-envoy\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.390434 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.390436    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-credential-socket\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.390706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.390494    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-token\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-token\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.390706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.390638    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-kube-api-access-mzlcs\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-kube-api-access-mzlcs\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.390822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.390809    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-data\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.390881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.390868    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-certs\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.390945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.390801    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-credential-socket" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.390945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.390915    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-podinfo\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.391058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391000    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a451d097-a43a-42eb-beaa-e7b394e8efb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a451d097-a43a-42eb-beaa-e7b394e8efb9-istiod-ca-cert\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.391123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391096    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-data" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.391186 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391124    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-certs" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.391360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391329    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-socket\") pod \"a451d097-a43a-42eb-beaa-e7b394e8efb9\" (UID: \"a451d097-a43a-42eb-beaa-e7b394e8efb9\") "
Oct 30 22:50:17.391449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391373    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a451d097-a43a-42eb-beaa-e7b394e8efb9-istiod-ca-cert" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:17.391661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391641    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-socket" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.391911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391894    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.391990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391913    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.391990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391924    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.391990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391933    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.391990 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.391954    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a451d097-a43a-42eb-beaa-e7b394e8efb9-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.393764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.393732    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-envoy" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.393908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.393786    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-token" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:17.393908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.393821    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-kube-api-access-mzlcs" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "kube-api-access-mzlcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:17.395482 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.395456    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-podinfo" pod "a451d097-a43a-42eb-beaa-e7b394e8efb9" (UID: "a451d097-a43a-42eb-beaa-e7b394e8efb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:17.493233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.493184    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.493233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.493234    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.493510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.493245    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzlcs\" (UniqueName: \"kubernetes.io/projected/a451d097-a43a-42eb-beaa-e7b394e8efb9-kube-api-access-mzlcs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.493510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:17.493258    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a451d097-a43a-42eb-beaa-e7b394e8efb9-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:18.058889 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.058856    4581 generic.go:356] "Generic (PLEG): container finished" podID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerID="7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6" exitCode=0
Oct 30 22:50:18.059102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.058931    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"
Oct 30 22:50:18.059102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.058941    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" event={"ID":"a451d097-a43a-42eb-beaa-e7b394e8efb9","Type":"ContainerDied","Data":"7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6"}
Oct 30 22:50:18.059102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.059012    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx" event={"ID":"a451d097-a43a-42eb-beaa-e7b394e8efb9","Type":"ContainerDied","Data":"c23658f0644e5e1a8adec4a596f2c276c21df3b45a0c32637d2cff87d0928a78"}
Oct 30 22:50:18.059102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.059046    4581 scope.go:122] "RemoveContainer" containerID="7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6"
Oct 30 22:50:18.079290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.079262    4581 scope.go:122] "RemoveContainer" containerID="798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c"
Oct 30 22:50:18.091867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.091848    4581 scope.go:122] "RemoveContainer" containerID="7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6"
Oct 30 22:50:18.092261 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:50:18.092236    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6\": container with ID starting with 7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6 not found: ID does not exist" containerID="7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6"
Oct 30 22:50:18.092422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.092272    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6"} err="failed to get container status \"7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6\": rpc error: code = NotFound desc = could not find container \"7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6\": container with ID starting with 7065905d481ddff4ed126570311543371eb6c623cf16b3dcb999343aa06c9ab6 not found: ID does not exist"
Oct 30 22:50:18.092422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.092295    4581 scope.go:122] "RemoveContainer" containerID="798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c"
Oct 30 22:50:18.092630 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:50:18.092601    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c\": container with ID starting with 798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c not found: ID does not exist" containerID="798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c"
Oct 30 22:50:18.092689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.092641    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c"} err="failed to get container status \"798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c\": rpc error: code = NotFound desc = could not find container \"798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c\": container with ID starting with 798bf9916e71ff088fdb1b15cf7a70ad139d20a9197ca193e1bae176c21f422c not found: ID does not exist"
Oct 30 22:50:18.118707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.118672    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"]
Oct 30 22:50:18.123211 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:18.123175    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-jlqnx"]
Oct 30 22:50:19.356529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.356497    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"]
Oct 30 22:50:19.357201 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.357184    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy"
Oct 30 22:50:19.357298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.357202    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df7ae0f-f6b7-4a5d-bfd8-1faec99eebf2" containerName="istio-proxy"
Oct 30 22:50:19.357298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.357239    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy"
Oct 30 22:50:19.357298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.357245    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy"
Oct 30 22:50:19.357298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.357254    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy"
Oct 30 22:50:19.357298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.357259    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy"
Oct 30 22:50:19.357469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.357397    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy"
Oct 30 22:50:19.357469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.357405    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" containerName="istio-proxy"
Oct 30 22:50:19.362443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.362421    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:19.382011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.381978    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"]
Oct 30 22:50:19.384940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.384911    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cdc122\""
Oct 30 22:50:19.433288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.417382    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18be17e-6d08-4be0-a6df-7f4f8950214e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4\" (UID: \"d18be17e-6d08-4be0-a6df-7f4f8950214e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:19.453246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.433634    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2ns\" (UniqueName: \"kubernetes.io/projected/d18be17e-6d08-4be0-a6df-7f4f8950214e-kube-api-access-ss2ns\") pod \"payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4\" (UID: \"d18be17e-6d08-4be0-a6df-7f4f8950214e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:19.453615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.453570    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cdc122-dockercfg-prrvz\""
Oct 30 22:50:19.556298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.556259    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18be17e-6d08-4be0-a6df-7f4f8950214e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4\" (UID: \"d18be17e-6d08-4be0-a6df-7f4f8950214e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:19.556604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.556582    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2ns\" (UniqueName: \"kubernetes.io/projected/d18be17e-6d08-4be0-a6df-7f4f8950214e-kube-api-access-ss2ns\") pod \"payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4\" (UID: \"d18be17e-6d08-4be0-a6df-7f4f8950214e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:19.558129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.558102    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18be17e-6d08-4be0-a6df-7f4f8950214e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4\" (UID: \"d18be17e-6d08-4be0-a6df-7f4f8950214e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:19.622427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.622353    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2ns\" (UniqueName: \"kubernetes.io/projected/d18be17e-6d08-4be0-a6df-7f4f8950214e-kube-api-access-ss2ns\") pod \"payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4\" (UID: \"d18be17e-6d08-4be0-a6df-7f4f8950214e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:19.680954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.680914    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:19.996194 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:19.996157    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"]
Oct 30 22:50:19.998202 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:50:19.998175    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18be17e_6d08_4be0_a6df_7f4f8950214e.slice/crio-b425c9bc7acfc09c5b25c770abcb1320dfe3ba7c09a235bc99eff1d2add50201 WatchSource:0}: Error finding container b425c9bc7acfc09c5b25c770abcb1320dfe3ba7c09a235bc99eff1d2add50201: Status 404 returned error can't find the container with id b425c9bc7acfc09c5b25c770abcb1320dfe3ba7c09a235bc99eff1d2add50201
Oct 30 22:50:20.083325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:20.083195    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4" event={"ID":"d18be17e-6d08-4be0-a6df-7f4f8950214e","Type":"ContainerStarted","Data":"b425c9bc7acfc09c5b25c770abcb1320dfe3ba7c09a235bc99eff1d2add50201"}
Oct 30 22:50:20.098748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:20.098701    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a451d097-a43a-42eb-beaa-e7b394e8efb9" path="/var/lib/kubelet/pods/a451d097-a43a-42eb-beaa-e7b394e8efb9/volumes"
Oct 30 22:50:21.094506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:21.094473    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4" event={"ID":"d18be17e-6d08-4be0-a6df-7f4f8950214e","Type":"ContainerStarted","Data":"58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196"}
Oct 30 22:50:21.095045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:21.094574    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:21.129599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:21.129551    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4" podStartSLOduration=2.129538005 podStartE2EDuration="2.129538005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:21.127911402 +0000 UTC m=+2605.166850288" watchObservedRunningTime="2026-10-30 22:50:21.129538005 +0000 UTC m=+2605.168476891"
Oct 30 22:50:28.357945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.357911    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"]
Oct 30 22:50:28.362518 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.362501    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.385822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.385458    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:28.385822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.385559    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:28.385822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.385557    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:28.385822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.385647    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:50:28.391642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.391618    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"]
Oct 30 22:50:28.429246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.422657    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.429653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.429621    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtj5\" (UniqueName: \"kubernetes.io/projected/3d8659f1-6981-4149-929b-6dc69fafc3a8-kube-api-access-xjtj5\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.433253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.429953    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.542982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.542947    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.542982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.542989    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtj5\" (UniqueName: \"kubernetes.io/projected/3d8659f1-6981-4149-929b-6dc69fafc3a8-kube-api-access-xjtj5\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.543273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.543059    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.543672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.543651    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.545665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.545645    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.560389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.560357    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtj5\" (UniqueName: \"kubernetes.io/projected/3d8659f1-6981-4149-929b-6dc69fafc3a8-kube-api-access-xjtj5\") pod \"maas-api-revoke-sub-e56820a1-qdbp5\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.678906 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.678819    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:28.989838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:28.989808    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"]
Oct 30 22:50:28.990966 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:50:28.990941    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8659f1_6981_4149_929b_6dc69fafc3a8.slice/crio-ed58ec8e7630ad780d15c46c62e650e3352d522b9df5b6601a0f253f974b785a WatchSource:0}: Error finding container ed58ec8e7630ad780d15c46c62e650e3352d522b9df5b6601a0f253f974b785a: Status 404 returned error can't find the container with id ed58ec8e7630ad780d15c46c62e650e3352d522b9df5b6601a0f253f974b785a
Oct 30 22:50:29.203848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:29.203807    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5" event={"ID":"3d8659f1-6981-4149-929b-6dc69fafc3a8","Type":"ContainerStarted","Data":"ed58ec8e7630ad780d15c46c62e650e3352d522b9df5b6601a0f253f974b785a"}
Oct 30 22:50:30.213019 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:30.212919    4581 generic.go:356] "Generic (PLEG): container finished" podID="3d8659f1-6981-4149-929b-6dc69fafc3a8" containerID="40e2cbddc3036a89d2892b78c35f0f808d853be2104387eb91b91cf865d2a2cc" exitCode=0
Oct 30 22:50:30.213412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:30.213115    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5" event={"ID":"3d8659f1-6981-4149-929b-6dc69fafc3a8","Type":"ContainerDied","Data":"40e2cbddc3036a89d2892b78c35f0f808d853be2104387eb91b91cf865d2a2cc"}
Oct 30 22:50:31.390200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.390167    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:31.488711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.488608    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-openshift-ca\") pod \"3d8659f1-6981-4149-929b-6dc69fafc3a8\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") "
Oct 30 22:50:31.488957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.488938    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8659f1-6981-4149-929b-6dc69fafc3a8-kube-api-access-xjtj5\" (UniqueName: \"kubernetes.io/projected/3d8659f1-6981-4149-929b-6dc69fafc3a8-kube-api-access-xjtj5\") pod \"3d8659f1-6981-4149-929b-6dc69fafc3a8\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") "
Oct 30 22:50:31.489033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.488999    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-xks-ca\") pod \"3d8659f1-6981-4149-929b-6dc69fafc3a8\" (UID: \"3d8659f1-6981-4149-929b-6dc69fafc3a8\") "
Oct 30 22:50:31.489656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.489623    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-openshift-ca" pod "3d8659f1-6981-4149-929b-6dc69fafc3a8" (UID: "3d8659f1-6981-4149-929b-6dc69fafc3a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:31.492313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.492286    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8659f1-6981-4149-929b-6dc69fafc3a8-kube-api-access-xjtj5" pod "3d8659f1-6981-4149-929b-6dc69fafc3a8" (UID: "3d8659f1-6981-4149-929b-6dc69fafc3a8"). InnerVolumeSpecName "kube-api-access-xjtj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:31.492430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.492368    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-xks-ca" pod "3d8659f1-6981-4149-929b-6dc69fafc3a8" (UID: "3d8659f1-6981-4149-929b-6dc69fafc3a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:31.591505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.591467    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjtj5\" (UniqueName: \"kubernetes.io/projected/3d8659f1-6981-4149-929b-6dc69fafc3a8-kube-api-access-xjtj5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:31.591505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.591503    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:31.591763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:31.591517    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8659f1-6981-4149-929b-6dc69fafc3a8-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:32.107066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:32.107039    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 22:50:32.231096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:32.231069    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"
Oct 30 22:50:32.231323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:32.231107    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5" event={"ID":"3d8659f1-6981-4149-929b-6dc69fafc3a8","Type":"ContainerDied","Data":"ed58ec8e7630ad780d15c46c62e650e3352d522b9df5b6601a0f253f974b785a"}
Oct 30 22:50:32.231323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:32.231164    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed58ec8e7630ad780d15c46c62e650e3352d522b9df5b6601a0f253f974b785a"
Oct 30 22:50:38.695443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.695396    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"]
Oct 30 22:50:38.696461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.696437    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3d8659f1-6981-4149-929b-6dc69fafc3a8" containerName="revoke-keys"
Oct 30 22:50:38.696596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.696462    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8659f1-6981-4149-929b-6dc69fafc3a8" containerName="revoke-keys"
Oct 30 22:50:38.696666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.696637    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8659f1-6981-4149-929b-6dc69fafc3a8" containerName="revoke-keys"
Oct 30 22:50:38.701131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.701104    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.714036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.714004    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.714200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.714153    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fnd\" (UniqueName: \"kubernetes.io/projected/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-kube-api-access-24fnd\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.714385 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.714298    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.719378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.719351    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"]
Oct 30 22:50:38.721963 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.721942    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:38.722066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.722044    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:38.722978 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.722944    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:50:38.723090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.723035    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:38.815769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.815723    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.815983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.815824    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fnd\" (UniqueName: \"kubernetes.io/projected/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-kube-api-access-24fnd\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.815983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.815943    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.816566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.816547    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.818277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.818255    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:38.834876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:38.834845    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fnd\" (UniqueName: \"kubernetes.io/projected/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-kube-api-access-24fnd\") pod \"maas-api-revoke-sub-42e0bf3e-rt87m\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:39.021420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:39.021334    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:39.328315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:39.328281    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"]
Oct 30 22:50:39.329816 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:50:39.329794    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f176d1_11ae_4a50_94fa_c5a0e6cf8a9e.slice/crio-51223586326b4ead28006724ccaa0c901ee08b61fda7303db3193fbe0b0bb8eb WatchSource:0}: Error finding container 51223586326b4ead28006724ccaa0c901ee08b61fda7303db3193fbe0b0bb8eb: Status 404 returned error can't find the container with id 51223586326b4ead28006724ccaa0c901ee08b61fda7303db3193fbe0b0bb8eb
Oct 30 22:50:40.310699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:40.310654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m" event={"ID":"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e","Type":"ContainerStarted","Data":"4c2f9f17f3b525ddfdf96551901fc4001ce69e5f907422d24eb778b9709dfaa2"}
Oct 30 22:50:40.311275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:40.310710    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m" event={"ID":"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e","Type":"ContainerStarted","Data":"51223586326b4ead28006724ccaa0c901ee08b61fda7303db3193fbe0b0bb8eb"}
Oct 30 22:50:40.348238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:40.347399    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m" podStartSLOduration=2.347381687 podStartE2EDuration="2.347381687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:40.344956696 +0000 UTC m=+2624.383895587" watchObservedRunningTime="2026-10-30 22:50:40.347381687 +0000 UTC m=+2624.386320572"
Oct 30 22:50:40.819242 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:50:40.819177    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:50:41.320565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:41.320524    4581 generic.go:356] "Generic (PLEG): container finished" podID="b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e" containerID="4c2f9f17f3b525ddfdf96551901fc4001ce69e5f907422d24eb778b9709dfaa2" exitCode=0
Oct 30 22:50:41.320969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:41.320609    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m" event={"ID":"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e","Type":"ContainerDied","Data":"4c2f9f17f3b525ddfdf96551901fc4001ce69e5f907422d24eb778b9709dfaa2"}
Oct 30 22:50:42.488040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.488011    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:42.565586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.565551    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-kube-api-access-24fnd\" (UniqueName: \"kubernetes.io/projected/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-kube-api-access-24fnd\") pod \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") "
Oct 30 22:50:42.565764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.565609    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-openshift-ca\") pod \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") "
Oct 30 22:50:42.565764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.565696    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-xks-ca\") pod \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\" (UID: \"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e\") "
Oct 30 22:50:42.566083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.566050    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-openshift-ca" pod "b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e" (UID: "b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:42.566399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.566375    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:42.568270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.568209    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-xks-ca" pod "b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e" (UID: "b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:42.568429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.568314    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-kube-api-access-24fnd" pod "b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e" (UID: "b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e"). InnerVolumeSpecName "kube-api-access-24fnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:42.667713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.667684    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fnd\" (UniqueName: \"kubernetes.io/projected/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-kube-api-access-24fnd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:42.667713 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:42.667710    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:50:43.351519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:43.351490    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"
Oct 30 22:50:43.351732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:43.351523    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m" event={"ID":"b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e","Type":"ContainerDied","Data":"51223586326b4ead28006724ccaa0c901ee08b61fda7303db3193fbe0b0bb8eb"}
Oct 30 22:50:43.351732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:43.351570    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51223586326b4ead28006724ccaa0c901ee08b61fda7303db3193fbe0b0bb8eb"
Oct 30 22:50:54.052583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:54.052543    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"]
Oct 30 22:50:54.061128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:54.061096    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be69375a-dxwts"]
Oct 30 22:50:54.099730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:50:54.099699    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d85666-683f-40bd-b9b8-c16f9a5a4cfe" path="/var/lib/kubelet/pods/63d85666-683f-40bd-b9b8-c16f9a5a4cfe/volumes"
Oct 30 22:51:05.030581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:05.030544    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"]
Oct 30 22:51:05.039256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:05.039226    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4a3937-xcp5c"]
Oct 30 22:51:06.100290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:06.100258    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80379a74-57ca-4d4e-87a7-0336960df0a6" path="/var/lib/kubelet/pods/80379a74-57ca-4d4e-87a7-0336960df0a6/volumes"
Oct 30 22:51:10.867243 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:51:10.859828    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:51:18.709900 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.709861    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"]
Oct 30 22:51:18.710998 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.710974    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e" containerName="revoke-keys"
Oct 30 22:51:18.711150 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.711000    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e" containerName="revoke-keys"
Oct 30 22:51:18.711239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.711189    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e" containerName="revoke-keys"
Oct 30 22:51:18.721908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.721832    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.738325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.738283    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e3a7dd-openshift-default-dockercfg-tc7ht\""
Oct 30 22:51:18.741535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.740958    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"]
Oct 30 22:51:18.802352 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.802316    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkl8s\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-kube-api-access-lkl8s\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.802677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.802634    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-socket\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.802849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.802831    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b658db4-7498-4462-850d-ba2669a04dac-istio-podinfo\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.803141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.803122    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-envoy\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.803357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.803325    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-istio-token\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.803529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.803509    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-data\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.803689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.803668    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-credential-socket\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.803841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.803824    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b658db4-7498-4462-850d-ba2669a04dac-istiod-ca-cert\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.804042 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.804009    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-certs\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907186    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-envoy\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907284    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-istio-token\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907314    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-data\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907343    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-credential-socket\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907370    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b658db4-7498-4462-850d-ba2669a04dac-istiod-ca-cert\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907417    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-certs\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907573    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkl8s\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-kube-api-access-lkl8s\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907614    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-socket\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907637    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b658db4-7498-4462-850d-ba2669a04dac-istio-podinfo\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.908166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.907972    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-data\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.924239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.915949    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-certs\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.924239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.916313    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-credential-socket\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.924239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.916805    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-socket\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.939242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.934908    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b658db4-7498-4462-850d-ba2669a04dac-istiod-ca-cert\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.953411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.948390    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-envoy\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.959240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.953239    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b658db4-7498-4462-850d-ba2669a04dac-istio-podinfo\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.964166 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.962951    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-istio-token\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:18.987240 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:18.986861    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkl8s\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-kube-api-access-lkl8s\") pod \"e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") " pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:19.066253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:19.066160    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:19.440285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:19.440254    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"]
Oct 30 22:51:19.441652 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:51:19.441611    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b658db4_7498_4462_850d_ba2669a04dac.slice/crio-d9864da6e53b62c16b89fda45aa248a81716871231f58b64ee3a968b70db55d7 WatchSource:0}: Error finding container d9864da6e53b62c16b89fda45aa248a81716871231f58b64ee3a968b70db55d7: Status 404 returned error can't find the container with id d9864da6e53b62c16b89fda45aa248a81716871231f58b64ee3a968b70db55d7
Oct 30 22:51:19.444827 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:19.444782    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:19.444918 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:19.444874    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:19.444961 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:19.444919    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:19.747046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:19.746949    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" event={"ID":"2b658db4-7498-4462-850d-ba2669a04dac","Type":"ContainerStarted","Data":"4e4cd8865cd0d83a0d9d70e64741bcd27ca53265a310f62a09e088a1fca8b60a"}
Oct 30 22:51:19.747676 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:19.747632    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" event={"ID":"2b658db4-7498-4462-850d-ba2669a04dac","Type":"ContainerStarted","Data":"d9864da6e53b62c16b89fda45aa248a81716871231f58b64ee3a968b70db55d7"}
Oct 30 22:51:19.788322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:19.787602    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" podStartSLOduration=1.787557461 podStartE2EDuration="1.787557461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:19.785699345 +0000 UTC m=+2663.824638230" watchObservedRunningTime="2026-10-30 22:51:19.787557461 +0000 UTC m=+2663.826496348"
Oct 30 22:51:20.066460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:20.066363    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:20.073371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:20.073342    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:20.759366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:20.759324    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:51:20.760080 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:20.759387    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:51:20.760951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:20.760925    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:20.762149 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:20.762117    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:51:21.775376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:21.775290    4581 generic.go:356] "Generic (PLEG): container finished" podID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerID="c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b" exitCode=0
Oct 30 22:51:21.775815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:21.775377    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" event={"ID":"e2754d3d-bedb-443c-95a2-db9070f4ed02","Type":"ContainerDied","Data":"c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b"}
Oct 30 22:51:21.776035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:21.776017    4581 scope.go:122] "RemoveContainer" containerID="c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b"
Oct 30 22:51:21.777682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:21.777649    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:21.777796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:21.777720    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:21.777796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:21.777753    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:22.795940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:22.795904    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" event={"ID":"e2754d3d-bedb-443c-95a2-db9070f4ed02","Type":"ContainerStarted","Data":"857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6"}
Oct 30 22:51:22.831387 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:22.831351    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:51:22.831595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:22.831532    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 22:51:22.831595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:22.831576    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 22:51:23.831607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:23.831557    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 22:51:23.832397 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:23.832356    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 22:51:24.398797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.398761    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"]
Oct 30 22:51:24.405823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.405792    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"]
Oct 30 22:51:24.405985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.405907    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:24.427519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.427491    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e3a7dd-dockercfg-z2spv\""
Oct 30 22:51:24.427642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.427558    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e3a7dd\""
Oct 30 22:51:24.493911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.493873    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpcg\" (UniqueName: \"kubernetes.io/projected/1a0e9922-8e85-4425-9fbd-ec869b5e382e-kube-api-access-8vpcg\") pod \"payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2\" (UID: \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:24.494130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.494049    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0e9922-8e85-4425-9fbd-ec869b5e382e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2\" (UID: \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:24.595244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.595177    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0e9922-8e85-4425-9fbd-ec869b5e382e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2\" (UID: \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:24.595456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.595374    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpcg\" (UniqueName: \"kubernetes.io/projected/1a0e9922-8e85-4425-9fbd-ec869b5e382e-kube-api-access-8vpcg\") pod \"payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2\" (UID: \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:24.596077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.596054    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0e9922-8e85-4425-9fbd-ec869b5e382e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2\" (UID: \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:24.613602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.613575    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpcg\" (UniqueName: \"kubernetes.io/projected/1a0e9922-8e85-4425-9fbd-ec869b5e382e-kube-api-access-8vpcg\") pod \"payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2\" (UID: \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:24.720613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.720514    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:24.850399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.850360    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 22:51:24.850974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:24.850937    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 22:51:25.088000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:25.087964    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"]
Oct 30 22:51:25.091570 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:51:25.091544    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0e9922_8e85_4425_9fbd_ec869b5e382e.slice/crio-10a2a428d957ea378f9db6f864de0612b0545ce7835136dccf7eba8f24e7631d WatchSource:0}: Error finding container 10a2a428d957ea378f9db6f864de0612b0545ce7835136dccf7eba8f24e7631d: Status 404 returned error can't find the container with id 10a2a428d957ea378f9db6f864de0612b0545ce7835136dccf7eba8f24e7631d
Oct 30 22:51:25.832291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:25.832247    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 22:51:25.832494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:25.832347    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 22:51:25.924547 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:25.924504    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2" event={"ID":"1a0e9922-8e85-4425-9fbd-ec869b5e382e","Type":"ContainerStarted","Data":"b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5"}
Oct 30 22:51:25.924988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:25.924623    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2" event={"ID":"1a0e9922-8e85-4425-9fbd-ec869b5e382e","Type":"ContainerStarted","Data":"10a2a428d957ea378f9db6f864de0612b0545ce7835136dccf7eba8f24e7631d"}
Oct 30 22:51:25.924988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:25.924662    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:25.977165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:25.977095    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2" podStartSLOduration=1.977074617 podStartE2EDuration="1.977074617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:25.968670848 +0000 UTC m=+2670.007609734" watchObservedRunningTime="2026-10-30 22:51:25.977074617 +0000 UTC m=+2670.016013507"
Oct 30 22:51:26.831552 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:26.831507    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 22:51:26.831763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:26.831571    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 22:51:27.831502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:27.831463    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 22:51:27.831903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:27.831551    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 22:51:28.836390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:28.836354    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:51:28.836875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:28.836864    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:51:28.837730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:28.837708    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 22:51:33.760195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.760163    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"]
Oct 30 22:51:33.764765 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.764748    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.778288 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.778267    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6af86e-openshift-default-dockercfg-lgnvg\""
Oct 30 22:51:33.831849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.831808    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"]
Oct 30 22:51:33.876068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876030    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.876068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876073    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277811fe-0837-4fcf-8c7e-9e174c3bf630-istiod-ca-cert\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.876398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876100    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-podinfo\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.876398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876178    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9lff\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-kube-api-access-b9lff\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.876398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876375    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-certs\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.876532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876504    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-token\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.876636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876605    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-data\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.876762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876718    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-credential-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.876827 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.876776    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-envoy\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.978490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.978457    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.978697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.978502    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277811fe-0837-4fcf-8c7e-9e174c3bf630-istiod-ca-cert\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.978697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.978675    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-podinfo\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.978697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.978694    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lff\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-kube-api-access-b9lff\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.978965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.978761    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-certs\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.978965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.978831    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-token\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.978965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.978839    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.978965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.978952    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-data\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.979290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.979043    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-credential-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.979290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.979116    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-certs\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.979290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.979142    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-envoy\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.979447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.979351    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277811fe-0837-4fcf-8c7e-9e174c3bf630-istiod-ca-cert\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.979447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.979355    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-data\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.979447 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.979407    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-credential-socket\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.981365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.981333    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-envoy\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.981485 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.981463    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-podinfo\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.998725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.998699    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lff\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-kube-api-access-b9lff\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:33.998935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:33.998913    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-token\") pod \"e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") " pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:34.079117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:34.079020    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:34.405982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:34.405953    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"]
Oct 30 22:51:34.407046 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:51:34.407012    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277811fe_0837_4fcf_8c7e_9e174c3bf630.slice/crio-01610faf1612793d0b9add4a066195e300a5028f308619c9b44ae180196286b0 WatchSource:0}: Error finding container 01610faf1612793d0b9add4a066195e300a5028f308619c9b44ae180196286b0: Status 404 returned error can't find the container with id 01610faf1612793d0b9add4a066195e300a5028f308619c9b44ae180196286b0
Oct 30 22:51:34.410182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:34.410145    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:34.410334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:34.410276    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:34.410334 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:34.410324    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:51:35.025572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:35.025539    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" event={"ID":"277811fe-0837-4fcf-8c7e-9e174c3bf630","Type":"ContainerStarted","Data":"207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3"}
Oct 30 22:51:35.026028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:35.025595    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" event={"ID":"277811fe-0837-4fcf-8c7e-9e174c3bf630","Type":"ContainerStarted","Data":"01610faf1612793d0b9add4a066195e300a5028f308619c9b44ae180196286b0"}
Oct 30 22:51:35.087243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:35.074479    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" podStartSLOduration=2.074458239 podStartE2EDuration="2.074458239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:35.073720383 +0000 UTC m=+2679.112659277" watchObservedRunningTime="2026-10-30 22:51:35.074458239 +0000 UTC m=+2679.113397123"
Oct 30 22:51:35.114032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:35.114003    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:35.134489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:35.134455    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 22:51:35.134767 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:35.134737    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 22:51:36.080311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:36.080277    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 22:51:36.080726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:36.080339    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 22:51:36.936204 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:36.936173    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:51:37.079817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:37.079779    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 22:51:37.080033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:37.079845    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 22:51:38.090252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:38.088718    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 22:51:38.090252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:38.088781    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 22:51:39.081711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:39.081658    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 22:51:39.081902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:39.081728    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 22:51:40.079701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:40.079656    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 22:51:40.080360 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:40.080325    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 22:51:40.861804 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:51:40.861764    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:51:41.055346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.055309    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"]
Oct 30 22:51:41.061690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.061662    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:41.081714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.081688    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"]
Oct 30 22:51:41.083346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.082498    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e14e00\""
Oct 30 22:51:41.083346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.082578    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e14e00-dockercfg-zjs58\""
Oct 30 22:51:41.090598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.090575    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:41.102571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.102182    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:41.102841 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.102823    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 22:51:41.121735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.121623    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"]
Oct 30 22:51:41.231314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.231274    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:41.312467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.312394    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"]
Oct 30 22:51:41.320268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.317722    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac61f39-1903-445c-ad4a-ffa29553e058-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc\" (UID: \"aac61f39-1903-445c-ad4a-ffa29553e058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:41.320268 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.318078    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7hd\" (UniqueName: \"kubernetes.io/projected/aac61f39-1903-445c-ad4a-ffa29553e058-kube-api-access-cs7hd\") pod \"payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc\" (UID: \"aac61f39-1903-445c-ad4a-ffa29553e058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:41.420423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.420272    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1361806-d6d4-448e-94cf-8742c795b818-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb\" (UID: \"f1361806-d6d4-448e-94cf-8742c795b818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:41.420581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.420483    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdb4p\" (UniqueName: \"kubernetes.io/projected/f1361806-d6d4-448e-94cf-8742c795b818-kube-api-access-wdb4p\") pod \"payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb\" (UID: \"f1361806-d6d4-448e-94cf-8742c795b818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:41.420646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.420574    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac61f39-1903-445c-ad4a-ffa29553e058-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc\" (UID: \"aac61f39-1903-445c-ad4a-ffa29553e058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:41.420761 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.420739    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7hd\" (UniqueName: \"kubernetes.io/projected/aac61f39-1903-445c-ad4a-ffa29553e058-kube-api-access-cs7hd\") pod \"payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc\" (UID: \"aac61f39-1903-445c-ad4a-ffa29553e058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:41.421489 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.421466    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac61f39-1903-445c-ad4a-ffa29553e058-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc\" (UID: \"aac61f39-1903-445c-ad4a-ffa29553e058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:41.444604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.444570    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7hd\" (UniqueName: \"kubernetes.io/projected/aac61f39-1903-445c-ad4a-ffa29553e058-kube-api-access-cs7hd\") pod \"payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc\" (UID: \"aac61f39-1903-445c-ad4a-ffa29553e058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:41.522091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.522059    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1361806-d6d4-448e-94cf-8742c795b818-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb\" (UID: \"f1361806-d6d4-448e-94cf-8742c795b818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:41.522314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.522178    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdb4p\" (UniqueName: \"kubernetes.io/projected/f1361806-d6d4-448e-94cf-8742c795b818-kube-api-access-wdb4p\") pod \"payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb\" (UID: \"f1361806-d6d4-448e-94cf-8742c795b818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:41.522824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.522805    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1361806-d6d4-448e-94cf-8742c795b818-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb\" (UID: \"f1361806-d6d4-448e-94cf-8742c795b818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:41.542498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.542466    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdb4p\" (UniqueName: \"kubernetes.io/projected/f1361806-d6d4-448e-94cf-8742c795b818-kube-api-access-wdb4p\") pod \"payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb\" (UID: \"f1361806-d6d4-448e-94cf-8742c795b818\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:41.564581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.564550    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:41.697532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.693378    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:41.921960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:41.921895    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"]
Oct 30 22:51:41.923111 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:51:41.923088    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1361806_d6d4_448e_94cf_8742c795b818.slice/crio-2a7095e339582f39488809e54af89d7e2838cab3e8c7336c0a91e1e2006efe9a WatchSource:0}: Error finding container 2a7095e339582f39488809e54af89d7e2838cab3e8c7336c0a91e1e2006efe9a: Status 404 returned error can't find the container with id 2a7095e339582f39488809e54af89d7e2838cab3e8c7336c0a91e1e2006efe9a
Oct 30 22:51:42.045513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.045485    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"]
Oct 30 22:51:42.047357 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:51:42.047329    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac61f39_1903_445c_ad4a_ffa29553e058.slice/crio-e14cbc2b33041c56682d8827ed722ba89e23ed84faa640ff018c174642a5a87a WatchSource:0}: Error finding container e14cbc2b33041c56682d8827ed722ba89e23ed84faa640ff018c174642a5a87a: Status 404 returned error can't find the container with id e14cbc2b33041c56682d8827ed722ba89e23ed84faa640ff018c174642a5a87a
Oct 30 22:51:42.214920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.214822    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb" event={"ID":"f1361806-d6d4-448e-94cf-8742c795b818","Type":"ContainerStarted","Data":"4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80"}
Oct 30 22:51:42.214920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.214883    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb" event={"ID":"f1361806-d6d4-448e-94cf-8742c795b818","Type":"ContainerStarted","Data":"2a7095e339582f39488809e54af89d7e2838cab3e8c7336c0a91e1e2006efe9a"}
Oct 30 22:51:42.214920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.214913    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:42.216898 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.216865    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc" event={"ID":"aac61f39-1903-445c-ad4a-ffa29553e058","Type":"ContainerStarted","Data":"7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475"}
Oct 30 22:51:42.217025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.216919    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:42.217025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.216940    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc" event={"ID":"aac61f39-1903-445c-ad4a-ffa29553e058","Type":"ContainerStarted","Data":"e14cbc2b33041c56682d8827ed722ba89e23ed84faa640ff018c174642a5a87a"}
Oct 30 22:51:42.248290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.248200    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb" podStartSLOduration=1.248187035 podStartE2EDuration="1.248187035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.247618613 +0000 UTC m=+2686.286557499" watchObservedRunningTime="2026-10-30 22:51:42.248187035 +0000 UTC m=+2686.287125923"
Oct 30 22:51:42.282781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:42.282710    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc" podStartSLOduration=1.282692658 podStartE2EDuration="1.282692658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.279537127 +0000 UTC m=+2686.318476014" watchObservedRunningTime="2026-10-30 22:51:42.282692658 +0000 UTC m=+2686.321631545"
Oct 30 22:51:53.036656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:53.036622    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"]
Oct 30 22:51:53.044946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:53.044918    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ccb405-n954h"]
Oct 30 22:51:53.049623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:53.049600    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"]
Oct 30 22:51:53.055431 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:53.055406    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284bc491-mtvcm"]
Oct 30 22:51:53.229003 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:53.228971    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 22:51:53.229185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:53.229047    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 22:51:54.100504 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:54.100470    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3c5bee-4af9-4346-b452-f1501a45b0b6" path="/var/lib/kubelet/pods/5d3c5bee-4af9-4346-b452-f1501a45b0b6/volumes"
Oct 30 22:51:54.100995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:54.100980    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29" path="/var/lib/kubelet/pods/c21ea171-bbf6-47f1-ab4e-f7bd11e3cc29/volumes"
Oct 30 22:51:57.260102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:57.255407    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:51:57.266840 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:51:57.266819    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:52:01.168651 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:01.168619    4581 scope.go:122] "RemoveContainer" containerID="537ceb31f7e6b99dffcefedc4ccb088494d84da98b9b92356c0f5be2f488faa4"
Oct 30 22:52:04.036636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:04.036591    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"]
Oct 30 22:52:04.046021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:04.045970    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f8d403-sd78r"]
Oct 30 22:52:04.106267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:04.105656    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6fddfb-fdae-46bf-a2c3-91ca389c2e25" path="/var/lib/kubelet/pods/2e6fddfb-fdae-46bf-a2c3-91ca389c2e25/volumes"
Oct 30 22:52:05.032406 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:05.032365    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"]
Oct 30 22:52:05.044433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:05.044398    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7737f85d-269fx"]
Oct 30 22:52:06.098401 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:06.098368    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c431ab82-888a-485e-8942-30a9718b4fd4" path="/var/lib/kubelet/pods/c431ab82-888a-485e-8942-30a9718b4fd4/volumes"
Oct 30 22:52:10.835918 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:52:10.835877    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:52:13.551839 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.551798    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"]
Oct 30 22:52:13.570740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.567050    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.584717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.584688    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"]
Oct 30 22:52:13.728492 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.728455    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.728728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.728623    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.728728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.728667    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4483810-5f34-443d-937c-9e7354fa50ef-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.728728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.728711    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.729022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.728847    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.729022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.728876    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.729022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.728994    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4f5\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-kube-api-access-nz4f5\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.729311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.729092    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4483810-5f34-443d-937c-9e7354fa50ef-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.729311 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.729131    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830055    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830104    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4483810-5f34-443d-937c-9e7354fa50ef-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830136    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830161    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830177    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830246    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4f5\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-kube-api-access-nz4f5\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830287    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4483810-5f34-443d-937c-9e7354fa50ef-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830318    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830456    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830560    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830617    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830637    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.830837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.830727    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.831235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.831200    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4483810-5f34-443d-937c-9e7354fa50ef-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.832947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.832924    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4483810-5f34-443d-937c-9e7354fa50ef-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.833087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.832994    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.847486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.847460    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.847782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.847762    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4f5\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-kube-api-access-nz4f5\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:13.921523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:13.921481    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:14.265477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.265445    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"]
Oct 30 22:52:14.267071 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:52:14.267041    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4483810_5f34_443d_937c_9e7354fa50ef.slice/crio-ef2e4299dea0340ac065865c09a10af0e490e536e34f5bf21593548a6830a73b WatchSource:0}: Error finding container ef2e4299dea0340ac065865c09a10af0e490e536e34f5bf21593548a6830a73b: Status 404 returned error can't find the container with id ef2e4299dea0340ac065865c09a10af0e490e536e34f5bf21593548a6830a73b
Oct 30 22:52:14.270422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.270390    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:14.270507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.270455    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:14.270507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.270484    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:14.673804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.673768    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" event={"ID":"c4483810-5f34-443d-937c-9e7354fa50ef","Type":"ContainerStarted","Data":"0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d"}
Oct 30 22:52:14.674315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.673833    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" event={"ID":"c4483810-5f34-443d-937c-9e7354fa50ef","Type":"ContainerStarted","Data":"ef2e4299dea0340ac065865c09a10af0e490e536e34f5bf21593548a6830a73b"}
Oct 30 22:52:14.676414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.676383    4581 generic.go:356] "Generic (PLEG): container finished" podID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerID="3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d" exitCode=0
Oct 30 22:52:14.676543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.676471    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" event={"ID":"3edd8469-95e9-41ee-a592-9c6c20aaff03","Type":"ContainerDied","Data":"3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d"}
Oct 30 22:52:14.676981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.676957    4581 scope.go:122] "RemoveContainer" containerID="3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d"
Oct 30 22:52:14.678897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.678867    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:14.678981 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.678945    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:14.679028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.678985    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:14.735151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.735093    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" podStartSLOduration=1.7350743830000002 podStartE2EDuration="1.735074383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:14.723251529 +0000 UTC m=+2718.762190414" watchObservedRunningTime="2026-10-30 22:52:14.735074383 +0000 UTC m=+2718.774013272"
Oct 30 22:52:14.922510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:14.922473    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:15.718242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:15.717915    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" event={"ID":"3edd8469-95e9-41ee-a592-9c6c20aaff03","Type":"ContainerStarted","Data":"4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426"}
Oct 30 22:52:15.923941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:15.923911    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 22:52:15.924169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:15.924146    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 22:52:16.038876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:16.038793    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:52:16.541881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:16.541845    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:52:16.543001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:16.542974    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 22:52:16.543156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:16.543023    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 22:52:16.922789 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:16.922760    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 22:52:16.923290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:16.923262    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 22:52:17.542181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:17.542142    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 22:52:17.542422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:17.542200    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 22:52:17.922861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:17.922564    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 22:52:17.922861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:17.922639    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 22:52:18.542277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:18.542244    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 22:52:18.542461 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:18.542300    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 22:52:18.922679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:18.922643    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 22:52:18.922871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:18.922701    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 22:52:19.541566 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:19.541521    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 22:52:19.542051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:19.541587    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 22:52:19.922145 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:19.922108    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 22:52:19.922364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:19.922173    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 22:52:20.542242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:20.542196    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 22:52:20.542711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:20.542675    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 22:52:20.928327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:20.928294    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:20.928834 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:20.928812    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:20.929690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:20.929667    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 22:52:21.546673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:21.546641    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:52:21.547777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:21.547753    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:52:29.030856 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:29.030819    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"]
Oct 30 22:52:29.039784 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:29.039754    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c457f51-rrmh6"]
Oct 30 22:52:30.100673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:30.100638    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84a0652-844d-42d1-93b5-1c4cb5ac96cd" path="/var/lib/kubelet/pods/c84a0652-844d-42d1-93b5-1c4cb5ac96cd/volumes"
Oct 30 22:52:40.850438 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:52:40.850346    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:52:45.431586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.427372    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"]
Oct 30 22:52:45.436648 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.436618    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"]
Oct 30 22:52:45.436801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.436730    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.453247 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.449748    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a60e80-openshift-default-dockercfg-7s5gs\""
Oct 30 22:52:45.531716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.531680    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-podinfo\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.531716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.531730    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-data\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.531939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.531761    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-credential-socket\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.531939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.531874    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-certs\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.531939 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.531919    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-socket\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.532155 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.532136    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istiod-ca-cert\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.532208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.532169    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-envoy\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.532208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.532198    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn547\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-kube-api-access-jn547\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.532341 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.532329    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-token\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635198    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istiod-ca-cert\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635278    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-envoy\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635321    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn547\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-kube-api-access-jn547\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635353    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-token\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635403    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-podinfo\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635445    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-data\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635490    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-credential-socket\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635542    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-certs\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.635586    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-socket\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.640243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.636151    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-socket\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.651092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.651056    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-credential-socket\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.653364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.651614    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-data\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.653543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.651842    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-certs\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.662755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.662727    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istiod-ca-cert\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.688058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.666076    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"]
Oct 30 22:52:45.688331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.668920    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-envoy\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.695847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.695781    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-podinfo\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.729722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.719748    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-token\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.744812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.744773    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn547\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-kube-api-access-jn547\") pod \"e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") " pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:45.809875 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:45.809608    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:46.195250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:46.195196    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"]
Oct 30 22:52:46.205764 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:46.205726    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:46.205894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:46.205814    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:46.205894 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:46.205858    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:47.138153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:47.138117    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" event={"ID":"2f0b4562-7c42-4605-ae23-06de1c2a5cfd","Type":"ContainerStarted","Data":"f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e"}
Oct 30 22:52:47.138610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:47.138184    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" event={"ID":"2f0b4562-7c42-4605-ae23-06de1c2a5cfd","Type":"ContainerStarted","Data":"4764df4a9079d99b62e8f9c3cc2f2a274df2823d394bd9daaa173455b819d173"}
Oct 30 22:52:47.197561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:47.197487    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podStartSLOduration=2.197466982 podStartE2EDuration="2.197466982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:47.186698731 +0000 UTC m=+2751.225637617" watchObservedRunningTime="2026-10-30 22:52:47.197466982 +0000 UTC m=+2751.236405869"
Oct 30 22:52:47.812754 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:47.812717    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:47.819612 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:47.819584    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:48.148467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:48.148436    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:48.149296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:48.149275    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:52:49.888848 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:52:49.888814    4581 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b658db4_7498_4462_850d_ba2669a04dac.slice/crio-conmon-4e4cd8865cd0d83a0d9d70e64741bcd27ca53265a310f62a09e088a1fca8b60a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:52:50.185051 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:50.185010    4581 generic.go:356] "Generic (PLEG): container finished" podID="2b658db4-7498-4462-850d-ba2669a04dac" containerID="4e4cd8865cd0d83a0d9d70e64741bcd27ca53265a310f62a09e088a1fca8b60a" exitCode=0
Oct 30 22:52:50.185259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:50.185090    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" event={"ID":"2b658db4-7498-4462-850d-ba2669a04dac","Type":"ContainerDied","Data":"4e4cd8865cd0d83a0d9d70e64741bcd27ca53265a310f62a09e088a1fca8b60a"}
Oct 30 22:52:50.185657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:50.185641    4581 scope.go:122] "RemoveContainer" containerID="4e4cd8865cd0d83a0d9d70e64741bcd27ca53265a310f62a09e088a1fca8b60a"
Oct 30 22:52:50.187571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:50.187540    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:50.187682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:50.187609    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:50.187682 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:50.187643    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:50.689235 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:50.689187    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2" podUID="1a0e9922-8e85-4425-9fbd-ec869b5e382e" containerName="payload-processing" containerID="cri-o://b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5" gracePeriod=25
Oct 30 22:52:50.761653 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:50.761612    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:52:51.038828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.038436    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:52:51.187657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.183846    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0e9922-8e85-4425-9fbd-ec869b5e382e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0e9922-8e85-4425-9fbd-ec869b5e382e-plugins-config-volume\") pod \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\" (UID: \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\") "
Oct 30 22:52:51.187657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.184027    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0e9922-8e85-4425-9fbd-ec869b5e382e-kube-api-access-8vpcg\" (UniqueName: \"kubernetes.io/projected/1a0e9922-8e85-4425-9fbd-ec869b5e382e-kube-api-access-8vpcg\") pod \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\" (UID: \"1a0e9922-8e85-4425-9fbd-ec869b5e382e\") "
Oct 30 22:52:51.187957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.187886    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0e9922-8e85-4425-9fbd-ec869b5e382e-plugins-config-volume" pod "1a0e9922-8e85-4425-9fbd-ec869b5e382e" (UID: "1a0e9922-8e85-4425-9fbd-ec869b5e382e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:51.190985 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.190955    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0e9922-8e85-4425-9fbd-ec869b5e382e-kube-api-access-8vpcg" pod "1a0e9922-8e85-4425-9fbd-ec869b5e382e" (UID: "1a0e9922-8e85-4425-9fbd-ec869b5e382e"). InnerVolumeSpecName "kube-api-access-8vpcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:51.201878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.201857    4581 generic.go:356] "Generic (PLEG): container finished" podID="1a0e9922-8e85-4425-9fbd-ec869b5e382e" containerID="b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5" exitCode=0
Oct 30 22:52:51.201965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.201923    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"
Oct 30 22:52:51.201965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.201938    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2" event={"ID":"1a0e9922-8e85-4425-9fbd-ec869b5e382e","Type":"ContainerDied","Data":"b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5"}
Oct 30 22:52:51.202072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.201990    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2" event={"ID":"1a0e9922-8e85-4425-9fbd-ec869b5e382e","Type":"ContainerDied","Data":"10a2a428d957ea378f9db6f864de0612b0545ce7835136dccf7eba8f24e7631d"}
Oct 30 22:52:51.202072 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.202021    4581 scope.go:122] "RemoveContainer" containerID="b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5"
Oct 30 22:52:51.204849 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.204820    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" event={"ID":"2b658db4-7498-4462-850d-ba2669a04dac","Type":"ContainerStarted","Data":"97642a3e44417d17a30b89f573939bb7539f03fc959b7b71b073d10c0115553c"}
Oct 30 22:52:51.218638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.218617    4581 scope.go:122] "RemoveContainer" containerID="b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5"
Oct 30 22:52:51.218914 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:52:51.218893    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5\": container with ID starting with b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5 not found: ID does not exist" containerID="b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5"
Oct 30 22:52:51.218966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.218923    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5"} err="failed to get container status \"b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5\": rpc error: code = NotFound desc = could not find container \"b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5\": container with ID starting with b9b1c2d19d8f7199de5e3bb8fd1df3937693bf3ef65d5dfd17ad332b4a32daa5 not found: ID does not exist"
Oct 30 22:52:51.287301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.286453    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpcg\" (UniqueName: \"kubernetes.io/projected/1a0e9922-8e85-4425-9fbd-ec869b5e382e-kube-api-access-8vpcg\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:52:51.287301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.286497    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0e9922-8e85-4425-9fbd-ec869b5e382e-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:52:51.311615 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.311584    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"]
Oct 30 22:52:51.347646 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:51.347594    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-666fdd8fcc-qvds2"]
Oct 30 22:52:52.078292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:52.077568    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:52:52.106242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:52.098566    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 22:52:52.106242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:52.098634    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 22:52:52.147986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:52.147946    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0e9922-8e85-4425-9fbd-ec869b5e382e" path="/var/lib/kubelet/pods/1a0e9922-8e85-4425-9fbd-ec869b5e382e/volumes"
Oct 30 22:52:53.080245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:53.079683    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 22:52:53.080245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:53.079781    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 22:52:54.071251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:54.067608    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 22:52:54.071251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:54.067689    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 22:52:55.040943 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:55.040910    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"]
Oct 30 22:52:55.050173 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:55.050147    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae2484d-cvvhd"]
Oct 30 22:52:55.066666 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:55.066628    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 22:52:55.066949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:55.066921    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 22:52:56.075321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:56.075283    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 22:52:56.075935 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:56.075900    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 22:52:56.099703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:56.099666    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e2cbbd-e9fc-43b5-a683-b3901acb948e" path="/var/lib/kubelet/pods/78e2cbbd-e9fc-43b5-a683-b3901acb948e/volumes"
Oct 30 22:52:57.072120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:57.072085    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:52:57.072669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:57.072636    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:52:57.073308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:57.073286    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 22:52:58.886645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.886606    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"]
Oct 30 22:52:58.887335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.887319    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1a0e9922-8e85-4425-9fbd-ec869b5e382e" containerName="payload-processing"
Oct 30 22:52:58.887405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.887335    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0e9922-8e85-4425-9fbd-ec869b5e382e" containerName="payload-processing"
Oct 30 22:52:58.887464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.887453    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0e9922-8e85-4425-9fbd-ec869b5e382e" containerName="payload-processing"
Oct 30 22:52:58.891448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.891425    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.913263 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.913237    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5983ea-openshift-default-dockercfg-mtrdw\""
Oct 30 22:52:58.920416 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.920388    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"]
Oct 30 22:52:58.959004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.958966    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-certs\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.959251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.959046    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-envoy\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.959251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.959182    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glk4n\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-kube-api-access-glk4n\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.959414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.959266    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-token\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.959414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.959366    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-credential-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.959565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.959537    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-data\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.959658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.959636    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-podinfo\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.959726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.959672    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istiod-ca-cert\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:58.959726 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:58.959705    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061277    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-podinfo\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061308 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061314    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istiod-ca-cert\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061336    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061478    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-certs\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061542    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-envoy\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061615    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glk4n\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-kube-api-access-glk4n\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061658    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-token\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061700    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-credential-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061777    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061826    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-certs\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.061870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.061841    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-data\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.062199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.062051    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-credential-socket\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.062199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.062094    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-data\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.062199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.062106    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istiod-ca-cert\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.063987 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.063956    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-envoy\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.064497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.064474    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-podinfo\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.078993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.078968    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-token\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.079117 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.079062    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glk4n\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-kube-api-access-glk4n\") pod \"e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") " pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.209522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.209440    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:52:59.560838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.560781    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"]
Oct 30 22:52:59.562332 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:52:59.562306    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998ef2e0_9da0_4c59_8d85_27fc27427cc3.slice/crio-49d9188a3b287aba38b5466ae14afd94f1d002b11631ff33dcce232828e52c0e WatchSource:0}: Error finding container 49d9188a3b287aba38b5466ae14afd94f1d002b11631ff33dcce232828e52c0e: Status 404 returned error can't find the container with id 49d9188a3b287aba38b5466ae14afd94f1d002b11631ff33dcce232828e52c0e
Oct 30 22:52:59.565191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.565156    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:59.565294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.565265    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:52:59.565343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:52:59.565300    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:53:00.288031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:00.287992    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" event={"ID":"998ef2e0-9da0-4c59-8d85-27fc27427cc3","Type":"ContainerStarted","Data":"f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d"}
Oct 30 22:53:00.288491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:00.288051    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" event={"ID":"998ef2e0-9da0-4c59-8d85-27fc27427cc3","Type":"ContainerStarted","Data":"49d9188a3b287aba38b5466ae14afd94f1d002b11631ff33dcce232828e52c0e"}
Oct 30 22:53:00.339793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:00.339733    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" podStartSLOduration=2.339712442 podStartE2EDuration="2.339712442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:00.334753454 +0000 UTC m=+2764.373692342" watchObservedRunningTime="2026-10-30 22:53:00.339712442 +0000 UTC m=+2764.378651333"
Oct 30 22:53:01.218251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:01.214946    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:53:01.224244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:01.223768    4581 scope.go:122] "RemoveContainer" containerID="a1f7fd00797673afbc889355506794665c06a895b20453b6248239899c6f8aed"
Oct 30 22:53:01.224244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:01.223988    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:53:01.224244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:01.224092    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 22:53:01.287465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:01.287439    4581 scope.go:122] "RemoveContainer" containerID="3ccefec74a77ca4e65b63e03fa85af09e7574095ce220082bfc96771ffabcaab"
Oct 30 22:53:01.302542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:01.302520    4581 scope.go:122] "RemoveContainer" containerID="f4e3c890d908390e375a474ab6a5ce538abd5159aac3a98083e2138735e47055"
Oct 30 22:53:01.313818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:01.313792    4581 scope.go:122] "RemoveContainer" containerID="f9c82c244c3c24cc399dfea36d9e5d73549e22f4f5e0a3be7da5319656dd1963"
Oct 30 22:53:01.325628 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:01.325123    4581 scope.go:122] "RemoveContainer" containerID="d2740b1fdfa69cb227705b2cc9196e6e14f54716e9da0a3098e233c9582f65ac"
Oct 30 22:53:02.210339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:02.210302    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:53:02.210523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:02.210374    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 22:53:03.210932 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:03.210893    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:53:03.211501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:03.210970    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 22:53:04.210364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:04.210328    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:53:04.210559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:04.210377    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 22:53:05.210412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:05.210353    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:53:05.210929 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:05.210455    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 22:53:06.246110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:06.246078    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:53:06.246692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:06.246668    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:53:06.247777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:06.247758    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 22:53:10.821431 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:53:10.821392    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:53:19.030971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:19.030934    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"]
Oct 30 22:53:19.038896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:19.038863    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b944c4-mttjv"]
Oct 30 22:53:20.101364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:20.101328    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0e4532-b323-4b58-be3f-fb4b80e1aeea" path="/var/lib/kubelet/pods/6e0e4532-b323-4b58-be3f-fb4b80e1aeea/volumes"
Oct 30 22:53:36.796770 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:36.796718    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"]
Oct 30 22:53:36.797284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:36.797005    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2" podUID="53888a7b-77ff-40c5-93ec-118de8225ffe" containerName="discovery" containerID="cri-o://b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857" gracePeriod=30
Oct 30 22:53:37.142525 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.142498    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:53:37.292944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.291444    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-dns-cert\") pod \"53888a7b-77ff-40c5-93ec-118de8225ffe\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") "
Oct 30 22:53:37.292944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.291520    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53888a7b-77ff-40c5-93ec-118de8225ffe-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53888a7b-77ff-40c5-93ec-118de8225ffe-local-certs\") pod \"53888a7b-77ff-40c5-93ec-118de8225ffe\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") "
Oct 30 22:53:37.292944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.291644    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-kubeconfig\") pod \"53888a7b-77ff-40c5-93ec-118de8225ffe\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") "
Oct 30 22:53:37.292944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.291687    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-kube-api-access-xhw7v\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-kube-api-access-xhw7v\") pod \"53888a7b-77ff-40c5-93ec-118de8225ffe\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") "
Oct 30 22:53:37.292944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.291723    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-ca-configmap\") pod \"53888a7b-77ff-40c5-93ec-118de8225ffe\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") "
Oct 30 22:53:37.292944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.291749    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-token\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-token\") pod \"53888a7b-77ff-40c5-93ec-118de8225ffe\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") "
Oct 30 22:53:37.292944 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.291779    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-cacerts\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-cacerts\") pod \"53888a7b-77ff-40c5-93ec-118de8225ffe\" (UID: \"53888a7b-77ff-40c5-93ec-118de8225ffe\") "
Oct 30 22:53:37.295750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.295696    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-ca-configmap" pod "53888a7b-77ff-40c5-93ec-118de8225ffe" (UID: "53888a7b-77ff-40c5-93ec-118de8225ffe"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:37.300279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.300229    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-token" pod "53888a7b-77ff-40c5-93ec-118de8225ffe" (UID: "53888a7b-77ff-40c5-93ec-118de8225ffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:37.316242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.307060    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-cacerts" pod "53888a7b-77ff-40c5-93ec-118de8225ffe" (UID: "53888a7b-77ff-40c5-93ec-118de8225ffe"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:37.316242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.307099    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53888a7b-77ff-40c5-93ec-118de8225ffe-local-certs" pod "53888a7b-77ff-40c5-93ec-118de8225ffe" (UID: "53888a7b-77ff-40c5-93ec-118de8225ffe"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:37.316242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.308641    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-kube-api-access-xhw7v" pod "53888a7b-77ff-40c5-93ec-118de8225ffe" (UID: "53888a7b-77ff-40c5-93ec-118de8225ffe"). InnerVolumeSpecName "kube-api-access-xhw7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:37.316242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.313753    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-kubeconfig" pod "53888a7b-77ff-40c5-93ec-118de8225ffe" (UID: "53888a7b-77ff-40c5-93ec-118de8225ffe"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:37.320546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.320340    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-dns-cert" pod "53888a7b-77ff-40c5-93ec-118de8225ffe" (UID: "53888a7b-77ff-40c5-93ec-118de8225ffe"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:37.393505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.393469    4581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-dns-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:53:37.393505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.393500    4581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53888a7b-77ff-40c5-93ec-118de8225ffe-local-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:53:37.393505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.393509    4581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-kubeconfig\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:53:37.393773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.393518    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhw7v\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-kube-api-access-xhw7v\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:53:37.393773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.393527    4581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-csr-ca-configmap\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:53:37.393773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.393535    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53888a7b-77ff-40c5-93ec-118de8225ffe-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:53:37.393773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.393544    4581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53888a7b-77ff-40c5-93ec-118de8225ffe-cacerts\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:53:37.868157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.868120    4581 generic.go:356] "Generic (PLEG): container finished" podID="53888a7b-77ff-40c5-93ec-118de8225ffe" containerID="b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857" exitCode=0
Oct 30 22:53:37.868675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.868169    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"
Oct 30 22:53:37.868675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.868209    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2" event={"ID":"53888a7b-77ff-40c5-93ec-118de8225ffe","Type":"ContainerDied","Data":"b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857"}
Oct 30 22:53:37.868675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.868310    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2" event={"ID":"53888a7b-77ff-40c5-93ec-118de8225ffe","Type":"ContainerDied","Data":"ac5c01e5156d494965662e19c70ddc3fbe5a482ba1eb701cf3fb3cd6cf441485"}
Oct 30 22:53:37.868675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.868336    4581 scope.go:122] "RemoveContainer" containerID="b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857"
Oct 30 22:53:37.888819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.888797    4581 scope.go:122] "RemoveContainer" containerID="b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857"
Oct 30 22:53:37.889242 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:53:37.889196    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857\": container with ID starting with b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857 not found: ID does not exist" containerID="b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857"
Oct 30 22:53:37.889377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.889265    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857"} err="failed to get container status \"b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857\": rpc error: code = NotFound desc = could not find container \"b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857\": container with ID starting with b48d7a68b393ae66d0d431715648cdf5f30cca28664de5e93f8ab8145f45e857 not found: ID does not exist"
Oct 30 22:53:37.917059 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.917030    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"]
Oct 30 22:53:37.926146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:37.926120    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47qt2"]
Oct 30 22:53:38.098236 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:38.098185    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53888a7b-77ff-40c5-93ec-118de8225ffe" path="/var/lib/kubelet/pods/53888a7b-77ff-40c5-93ec-118de8225ffe/volumes"
Oct 30 22:53:40.044075 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:40.044038    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"]
Oct 30 22:53:40.052665 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:40.052627    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3fc901-s6p68"]
Oct 30 22:53:40.099838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:40.099802    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8ad16c-b773-47fa-9c5d-7f583c5af9a6" path="/var/lib/kubelet/pods/3d8ad16c-b773-47fa-9c5d-7f583c5af9a6/volumes"
Oct 30 22:53:40.817451 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:53:40.817410    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:53:43.505699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.505656    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"]
Oct 30 22:53:43.506742 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.506718    4581 cpu_manager.go:414] "containerMap: removing container" podUID="53888a7b-77ff-40c5-93ec-118de8225ffe" containerName="discovery"
Oct 30 22:53:43.506843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.506746    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53888a7b-77ff-40c5-93ec-118de8225ffe" containerName="discovery"
Oct 30 22:53:43.507024 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.507006    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="53888a7b-77ff-40c5-93ec-118de8225ffe" containerName="discovery"
Oct 30 22:53:43.513014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.512986    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.536207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.536174    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e14e00\"/\"openshift-service-ca.crt\""
Oct 30 22:53:43.536207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.536176    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e14e00\"/\"default-dockercfg-4k7gl\""
Oct 30 22:53:43.536207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.536186    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e14e00\"/\"rate-test-model-e14e00-kserve-self-signed-certs\""
Oct 30 22:53:43.536490 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.536346    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e14e00\"/\"kube-root-ca.crt\""
Oct 30 22:53:43.540815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.540791    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"]
Oct 30 22:53:43.684190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.684154    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-tmp-dir\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.684386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.684305    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b465d7-ada9-4894-aa25-613cf1145ecf-tls-certs\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.684441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.684397    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-home\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.684534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.684505    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bp9f\" (UniqueName: \"kubernetes.io/projected/c4b465d7-ada9-4894-aa25-613cf1145ecf-kube-api-access-7bp9f\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.684641 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.684626    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-dshm\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.684747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.684730    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-model-cache\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.791609 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.791507    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-dshm\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.792561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.792534    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-model-cache\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.793706 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.793087    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-model-cache\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.793979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.793959    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-tmp-dir\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.794479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.794461    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b465d7-ada9-4894-aa25-613cf1145ecf-tls-certs\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.795545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.795379    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-home\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.795820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.795796    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bp9f\" (UniqueName: \"kubernetes.io/projected/c4b465d7-ada9-4894-aa25-613cf1145ecf-kube-api-access-7bp9f\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.807809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.798576    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-home\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.807809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.794370    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-tmp-dir\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.812339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.812300    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-dshm\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.828582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.828106    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b465d7-ada9-4894-aa25-613cf1145ecf-tls-certs\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:43.828582 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:43.828527    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bp9f\" (UniqueName: \"kubernetes.io/projected/c4b465d7-ada9-4894-aa25-613cf1145ecf-kube-api-access-7bp9f\") pod \"rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") " pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:44.128307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:44.128275    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:44.449694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:44.449651    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"]
Oct 30 22:53:44.451554 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:53:44.451513    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4b465d7_ada9_4894_aa25_613cf1145ecf.slice/crio-d0a395ed94ca947d1807a8cd94fc3e46efa7fa426eb739b4469adcf2d928828d WatchSource:0}: Error finding container d0a395ed94ca947d1807a8cd94fc3e46efa7fa426eb739b4469adcf2d928828d: Status 404 returned error can't find the container with id d0a395ed94ca947d1807a8cd94fc3e46efa7fa426eb739b4469adcf2d928828d
Oct 30 22:53:44.931817 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:44.931785    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv" event={"ID":"c4b465d7-ada9-4894-aa25-613cf1145ecf","Type":"ContainerStarted","Data":"21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47"}
Oct 30 22:53:44.932259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:44.931826    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv" event={"ID":"c4b465d7-ada9-4894-aa25-613cf1145ecf","Type":"ContainerStarted","Data":"d0a395ed94ca947d1807a8cd94fc3e46efa7fa426eb739b4469adcf2d928828d"}
Oct 30 22:53:44.971901 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:44.971849    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv" podStartSLOduration=1.971833137 podStartE2EDuration="1.971833137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:44.966311028 +0000 UTC m=+2809.005249914" watchObservedRunningTime="2026-10-30 22:53:44.971833137 +0000 UTC m=+2809.010772047"
Oct 30 22:53:45.128940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:45.128910    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:54.128689 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:54.128650    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:54.141740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:54.141714    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:53:55.037290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:53:55.037262    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:54:01.428530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:01.428500    4581 scope.go:122] "RemoveContainer" containerID="4526a65401193e9f26fbbbc8c3c0814097ae5b8fc7b7cb7e61cc933fd3aa0bdb"
Oct 30 22:54:01.439483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:01.439457    4581 scope.go:122] "RemoveContainer" containerID="38672501bde060c1d58ef89ea747889f6561b1a7387e8b45c80bdd6f2edb8a5b"
Oct 30 22:54:01.449780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:01.449757    4581 scope.go:122] "RemoveContainer" containerID="4cbbf0dcd417f2a3550d087337fa93379df193f3f3a855f41e2281b905838eed"
Oct 30 22:54:01.464400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:01.464371    4581 scope.go:122] "RemoveContainer" containerID="baf73e6aff93e58d7e696b7287cf0c3a09a6e6087ad21b8505b6abc8ac435cfc"
Oct 30 22:54:01.477752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:01.477731    4581 scope.go:122] "RemoveContainer" containerID="c0d7afa65581be7dabe579bbe3147616aa8c8dd04112dc85d0da2592e3565b4a"
Oct 30 22:54:01.488423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:01.488394    4581 scope.go:122] "RemoveContainer" containerID="015b1a1ec5c1d7d6fcb0a36dc06c2f81f5536b9055a73fe419958f392e2bc468"
Oct 30 22:54:01.500027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:01.500004    4581 scope.go:122] "RemoveContainer" containerID="e30412f2117afa533553d91ac1a05a80d06aa4353d2c8307518d4d68582e915d"
Oct 30 22:54:10.819914 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:54:10.819834    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:54:40.820463 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:54:40.820421    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:54:42.547811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:42.547725    4581 generic.go:356] "Generic (PLEG): container finished" podID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerID="f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e" exitCode=0
Oct 30 22:54:42.548285 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:42.547804    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" event={"ID":"2f0b4562-7c42-4605-ae23-06de1c2a5cfd","Type":"ContainerDied","Data":"f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e"}
Oct 30 22:54:42.548335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:42.548293    4581 scope.go:122] "RemoveContainer" containerID="f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e"
Oct 30 22:54:42.550144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:42.550112    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:54:42.550321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:42.550181    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:54:42.550321 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:42.550243    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:54:43.585242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:43.584740    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" event={"ID":"2f0b4562-7c42-4605-ae23-06de1c2a5cfd","Type":"ContainerStarted","Data":"a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c"}
Oct 30 22:54:43.812750 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:43.812716    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:54:43.814378 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:43.814352    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:54:43.814523 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:43.814401    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 22:54:44.813165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:44.813126    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:54:44.813632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:44.813199    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 22:54:45.812747 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:45.812711    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:54:45.812988 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:45.812812    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 22:54:46.812555 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:46.812515    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:54:46.812950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:46.812582    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 22:54:47.812716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:47.812679    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:54:47.813118 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:47.812749    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 22:54:48.149394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:48.149363    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:54:48.812500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:48.812468    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:54:48.812688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:48.812528    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 22:54:49.817772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:49.817742    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:54:49.819327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:54:49.819305    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 22:55:00.811049 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.807338    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"]
Oct 30 22:55:00.861683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.860889    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:00.869408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.869362    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"]
Oct 30 22:55:00.879045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.878999    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a60e80\""
Oct 30 22:55:00.900687 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.892046    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a60e80-dockercfg-zm8hf\""
Oct 30 22:55:00.903529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.903493    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"]
Oct 30 22:55:00.909821 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.909792    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:00.941399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.941361    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"]
Oct 30 22:55:00.970624 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.970584    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8bh9\" (UniqueName: \"kubernetes.io/projected/9142b111-3a39-4408-9784-52eda5706504-kube-api-access-m8bh9\") pod \"payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls\" (UID: \"9142b111-3a39-4408-9784-52eda5706504\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:00.970838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.970689    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286fb\" (UniqueName: \"kubernetes.io/projected/ae303102-96fd-484c-9bbf-ef20ed64ff25-kube-api-access-286fb\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw\" (UID: \"ae303102-96fd-484c-9bbf-ef20ed64ff25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:00.970838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.970799    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142b111-3a39-4408-9784-52eda5706504-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls\" (UID: \"9142b111-3a39-4408-9784-52eda5706504\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:00.970838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:00.970829    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae303102-96fd-484c-9bbf-ef20ed64ff25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw\" (UID: \"ae303102-96fd-484c-9bbf-ef20ed64ff25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:01.072304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.072120    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8bh9\" (UniqueName: \"kubernetes.io/projected/9142b111-3a39-4408-9784-52eda5706504-kube-api-access-m8bh9\") pod \"payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls\" (UID: \"9142b111-3a39-4408-9784-52eda5706504\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:01.072304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.072204    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286fb\" (UniqueName: \"kubernetes.io/projected/ae303102-96fd-484c-9bbf-ef20ed64ff25-kube-api-access-286fb\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw\" (UID: \"ae303102-96fd-484c-9bbf-ef20ed64ff25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:01.072563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.072338    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142b111-3a39-4408-9784-52eda5706504-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls\" (UID: \"9142b111-3a39-4408-9784-52eda5706504\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:01.072563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.072372    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae303102-96fd-484c-9bbf-ef20ed64ff25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw\" (UID: \"ae303102-96fd-484c-9bbf-ef20ed64ff25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:01.073610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.073399    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142b111-3a39-4408-9784-52eda5706504-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls\" (UID: \"9142b111-3a39-4408-9784-52eda5706504\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:01.073610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.073431    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae303102-96fd-484c-9bbf-ef20ed64ff25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw\" (UID: \"ae303102-96fd-484c-9bbf-ef20ed64ff25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:01.091532 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.091496    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8bh9\" (UniqueName: \"kubernetes.io/projected/9142b111-3a39-4408-9784-52eda5706504-kube-api-access-m8bh9\") pod \"payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls\" (UID: \"9142b111-3a39-4408-9784-52eda5706504\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:01.096432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.096370    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-286fb\" (UniqueName: \"kubernetes.io/projected/ae303102-96fd-484c-9bbf-ef20ed64ff25-kube-api-access-286fb\") pod \"payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw\" (UID: \"ae303102-96fd-484c-9bbf-ef20ed64ff25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:01.212952 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.212916    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:01.242102 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.242070    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:01.540233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.540188    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"]
Oct 30 22:55:01.542616 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:01.542572    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae303102_96fd_484c_9bbf_ef20ed64ff25.slice/crio-1d57ecd6afd68df2cb8fff8877b52efd9e3fed292d6cf7d74ff20bbfb0c88cf3 WatchSource:0}: Error finding container 1d57ecd6afd68df2cb8fff8877b52efd9e3fed292d6cf7d74ff20bbfb0c88cf3: Status 404 returned error can't find the container with id 1d57ecd6afd68df2cb8fff8877b52efd9e3fed292d6cf7d74ff20bbfb0c88cf3
Oct 30 22:55:01.544976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.544959    4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:55:01.576543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.576462    4581 scope.go:122] "RemoveContainer" containerID="f09624c14b347807ea0a4245bdd98ed4ee987ba9d5cd238318f8345bc6479f9d"
Oct 30 22:55:01.578572 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.578548    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"]
Oct 30 22:55:01.581121 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:01.581094    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9142b111_3a39_4408_9784_52eda5706504.slice/crio-c62503d7a24f2c67212a0943d1fad4bf26d5de93eacb56ea59c0b28b45609a8d WatchSource:0}: Error finding container c62503d7a24f2c67212a0943d1fad4bf26d5de93eacb56ea59c0b28b45609a8d: Status 404 returned error can't find the container with id c62503d7a24f2c67212a0943d1fad4bf26d5de93eacb56ea59c0b28b45609a8d
Oct 30 22:55:01.597983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.597954    4581 scope.go:122] "RemoveContainer" containerID="9a65836e07578d428651f794c743aaa01f6bb8cf46f95f2329248b61a55ceda8"
Oct 30 22:55:01.633309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.633278    4581 scope.go:122] "RemoveContainer" containerID="dfb481e60b08b1a94a93f4174fdbabc84c0a05f73b7cc67af9d1e20ef97c849d"
Oct 30 22:55:01.669921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.669894    4581 scope.go:122] "RemoveContainer" containerID="edbdab631ad4b182765e1ed2d64e928b95368e7e0d207586b1db55c7e1d78cfd"
Oct 30 22:55:01.791324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.791279    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw" event={"ID":"ae303102-96fd-484c-9bbf-ef20ed64ff25","Type":"ContainerStarted","Data":"1d57ecd6afd68df2cb8fff8877b52efd9e3fed292d6cf7d74ff20bbfb0c88cf3"}
Oct 30 22:55:01.793002 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:01.792962    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls" event={"ID":"9142b111-3a39-4408-9784-52eda5706504","Type":"ContainerStarted","Data":"c62503d7a24f2c67212a0943d1fad4bf26d5de93eacb56ea59c0b28b45609a8d"}
Oct 30 22:55:03.815740 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:03.815672    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw" event={"ID":"ae303102-96fd-484c-9bbf-ef20ed64ff25","Type":"ContainerStarted","Data":"2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b"}
Oct 30 22:55:03.816293 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:03.815789    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:03.818090 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:03.818053    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls" event={"ID":"9142b111-3a39-4408-9784-52eda5706504","Type":"ContainerStarted","Data":"ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2"}
Oct 30 22:55:03.818294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:03.818146    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:03.855024 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:03.854959    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw" podStartSLOduration=1.822437968 podStartE2EDuration="3.854939443s" totalImagesPullingTime="2.032501475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:03.851068474 +0000 UTC m=+2887.890007362" watchObservedRunningTime="2026-10-30 22:55:03.854939443 +0000 UTC m=+2887.893878333"
Oct 30 22:55:03.888038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:03.887971    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls" podStartSLOduration=1.86563546 podStartE2EDuration="3.887950605s" totalImagesPullingTime="2.022315145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:03.885099277 +0000 UTC m=+2887.924038158" watchObservedRunningTime="2026-10-30 22:55:03.887950605 +0000 UTC m=+2887.926889495"
Oct 30 22:55:10.819202 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:55:10.819155    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:55:14.831423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:14.831389    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 22:55:14.831870 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:14.831751    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 22:55:23.547038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:23.546973    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls_9142b111-3a39-4408-9784-52eda5706504/payload-processing/0.log"
Oct 30 22:55:24.915703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.915668    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"]
Oct 30 22:55:24.920114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.920095    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:24.934325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.934301    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:24.944013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.943987    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:24.944177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.944072    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:55:24.944177 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.944127    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:24.949806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.949773    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"]
Oct 30 22:55:24.963251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.953619    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:24.963251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.953807    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7sn\" (UniqueName: \"kubernetes.io/projected/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-kube-api-access-vh7sn\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:24.963251 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:24.953879    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:25.055517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:25.055456    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7sn\" (UniqueName: \"kubernetes.io/projected/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-kube-api-access-vh7sn\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:25.055734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:25.055602    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:25.055816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:25.055749    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:25.056577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:25.056552    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:25.058315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:25.058291    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:25.074100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:25.074071    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7sn\" (UniqueName: \"kubernetes.io/projected/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-kube-api-access-vh7sn\") pod \"maas-api-revoke-sub-976d46b7-lnxn5\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:25.233858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:25.233765    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:25.564357 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:25.564241    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"]
Oct 30 22:55:26.081316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.081274    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5" event={"ID":"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d","Type":"ContainerStarted","Data":"d93698fcf578c70d2edfff7ccb1f03d58c60eb65b0959e17a238328bddbdc8e6"}
Oct 30 22:55:26.081674 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.081345    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5" event={"ID":"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d","Type":"ContainerStarted","Data":"90fb6e0054432c2dfcd0768a7b188d0aa0a4e36bba677df78653d231e2d60cef"}
Oct 30 22:55:26.119781 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.118460    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5" podStartSLOduration=2.118441016 podStartE2EDuration="2.118441016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:26.116562677 +0000 UTC m=+2910.155501563" watchObservedRunningTime="2026-10-30 22:55:26.118441016 +0000 UTC m=+2910.157379903"
Oct 30 22:55:26.514639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.514606    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7281-2cbd22"]
Oct 30 22:55:26.519506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.519476    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:26.532688 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.532665    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xbw52\""
Oct 30 22:55:26.538524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.538490    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-2cbd22"]
Oct 30 22:55:26.577208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.577116    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"]
Oct 30 22:55:26.594571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.594532    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.604277 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.596682    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjxj\" (UniqueName: \"kubernetes.io/projected/b5271e81-ba3e-4fdd-9fbc-691894ac13bd-kube-api-access-jvjxj\") pod \"test-curl-7281-2cbd22\" (UID: \"b5271e81-ba3e-4fdd-9fbc-691894ac13bd\") " pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:26.621291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.621264    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"]
Oct 30 22:55:26.707323 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.707294    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx295\" (UniqueName: \"kubernetes.io/projected/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-kube-api-access-zx295\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.707550 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.707512    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.707664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.707609    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.707876 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.707857    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjxj\" (UniqueName: \"kubernetes.io/projected/b5271e81-ba3e-4fdd-9fbc-691894ac13bd-kube-api-access-jvjxj\") pod \"test-curl-7281-2cbd22\" (UID: \"b5271e81-ba3e-4fdd-9fbc-691894ac13bd\") " pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:26.726535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.726510    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjxj\" (UniqueName: \"kubernetes.io/projected/b5271e81-ba3e-4fdd-9fbc-691894ac13bd-kube-api-access-jvjxj\") pod \"test-curl-7281-2cbd22\" (UID: \"b5271e81-ba3e-4fdd-9fbc-691894ac13bd\") " pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:26.809763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.809654    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.809763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.809708    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.810026 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.809915    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx295\" (UniqueName: \"kubernetes.io/projected/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-kube-api-access-zx295\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.810983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.810476    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.814096 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.814048    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.828702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.828624    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx295\" (UniqueName: \"kubernetes.io/projected/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-kube-api-access-zx295\") pod \"maas-api-revoke-sub-ae9e4fe5-4hrjq\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:26.834455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.834424    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:26.926868 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:26.926613    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:27.097028 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.096935    4581 generic.go:356] "Generic (PLEG): container finished" podID="efde7b5e-b65f-450f-b47c-7e5fb76f8e1d" containerID="d93698fcf578c70d2edfff7ccb1f03d58c60eb65b0959e17a238328bddbdc8e6" exitCode=0
Oct 30 22:55:27.097453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.097038    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5" event={"ID":"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d","Type":"ContainerDied","Data":"d93698fcf578c70d2edfff7ccb1f03d58c60eb65b0959e17a238328bddbdc8e6"}
Oct 30 22:55:27.128458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.128430    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"]
Oct 30 22:55:27.133910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.133889    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.152888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.152852    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"]
Oct 30 22:55:27.163104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.163072    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-2cbd22"]
Oct 30 22:55:27.165657 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:27.165567    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5271e81_ba3e_4fdd_9fbc_691894ac13bd.slice/crio-fae6b9e88c5a7fe17f6069f115e360a0b45e954a8d361c1a2b332ccfa61b865c WatchSource:0}: Error finding container fae6b9e88c5a7fe17f6069f115e360a0b45e954a8d361c1a2b332ccfa61b865c: Status 404 returned error can't find the container with id fae6b9e88c5a7fe17f6069f115e360a0b45e954a8d361c1a2b332ccfa61b865c
Oct 30 22:55:27.219366 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.219337    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.219528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.219372    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2nxl\" (UniqueName: \"kubernetes.io/projected/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-kube-api-access-g2nxl\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.219613 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.219590    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.261208 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.261179    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"]
Oct 30 22:55:27.261870 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:27.261841    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc1a3ed_ff96_4d3d_86ba_4da37b7c3fd4.slice/crio-328a048976ca2d856ad4dada979945dbbf2ab492a519197ede23f06c5a894160 WatchSource:0}: Error finding container 328a048976ca2d856ad4dada979945dbbf2ab492a519197ede23f06c5a894160: Status 404 returned error can't find the container with id 328a048976ca2d856ad4dada979945dbbf2ab492a519197ede23f06c5a894160
Oct 30 22:55:27.320844 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.320817    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.321040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.321020    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.321171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.321052    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2nxl\" (UniqueName: \"kubernetes.io/projected/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-kube-api-access-g2nxl\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.321663 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.321639    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.323675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.323650    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.342258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.342200    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2nxl\" (UniqueName: \"kubernetes.io/projected/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-kube-api-access-g2nxl\") pod \"maas-api-revoke-sub-753b7a15-tcpbd\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.361455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.361425    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"]
Oct 30 22:55:27.368977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.368948    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.391270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.381857    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e3a7dd\"/\"default-dockercfg-8rrl5\""
Oct 30 22:55:27.391270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.382025    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e3a7dd\"/\"ipp-route-e3a7dd-kserve-self-signed-certs\""
Oct 30 22:55:27.391270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.382133    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e3a7dd\"/\"kube-root-ca.crt\""
Oct 30 22:55:27.391270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.382326    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e3a7dd\"/\"openshift-service-ca.crt\""
Oct 30 22:55:27.391641 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.391325    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"]
Oct 30 22:55:27.422858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.422810    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tls-certs\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.423120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.422874    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-dshm\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.423275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.423155    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdgd\" (UniqueName: \"kubernetes.io/projected/f8e2b178-91ce-4aa8-bf90-64b6066054f5-kube-api-access-4sdgd\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.423373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.423346    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-home\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.423448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.423371    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tmp-dir\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.423539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.423523    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-model-cache\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.457507 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.457476    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:27.525652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.524911    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdgd\" (UniqueName: \"kubernetes.io/projected/f8e2b178-91ce-4aa8-bf90-64b6066054f5-kube-api-access-4sdgd\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.525652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.525043    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-home\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.525652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.525077    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tmp-dir\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.525652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.525179    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-model-cache\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.525652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.525364    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tls-certs\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.525652 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.525423    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-dshm\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.526915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.526882    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tmp-dir\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.527745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.527568    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-home\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.530453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.530422    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-model-cache\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.531009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.530965    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-dshm\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.531727 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.531685    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tls-certs\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.545801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.545773    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdgd\" (UniqueName: \"kubernetes.io/projected/f8e2b178-91ce-4aa8-bf90-64b6066054f5-kube-api-access-4sdgd\") pod \"ipp-route-e3a7dd-kserve-788f9dbc89-5fg78\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") " pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.697814 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.697537    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:27.783971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:27.783935    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"]
Oct 30 22:55:27.787249 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:27.784650    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded08f5ea_cca4_4188_b81d_e2b2f8f5003b.slice/crio-a535d1eebec13506f517771ee26374fbda94cf8f62bfbf1f04ccfaab270f1072 WatchSource:0}: Error finding container a535d1eebec13506f517771ee26374fbda94cf8f62bfbf1f04ccfaab270f1072: Status 404 returned error can't find the container with id a535d1eebec13506f517771ee26374fbda94cf8f62bfbf1f04ccfaab270f1072
Oct 30 22:55:28.055956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.055932    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"]
Oct 30 22:55:28.058069 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:28.058041    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8e2b178_91ce_4aa8_bf90_64b6066054f5.slice/crio-c52a8cb5969836fa1aad80a02eee145d13161ac23e5f43e6a497258391b4861b WatchSource:0}: Error finding container c52a8cb5969836fa1aad80a02eee145d13161ac23e5f43e6a497258391b4861b: Status 404 returned error can't find the container with id c52a8cb5969836fa1aad80a02eee145d13161ac23e5f43e6a497258391b4861b
Oct 30 22:55:28.108169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.108137    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd" event={"ID":"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b","Type":"ContainerStarted","Data":"a535d1eebec13506f517771ee26374fbda94cf8f62bfbf1f04ccfaab270f1072"}
Oct 30 22:55:28.109679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.109653    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78" event={"ID":"f8e2b178-91ce-4aa8-bf90-64b6066054f5","Type":"ContainerStarted","Data":"c52a8cb5969836fa1aad80a02eee145d13161ac23e5f43e6a497258391b4861b"}
Oct 30 22:55:28.111655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.111608    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq" event={"ID":"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4","Type":"ContainerStarted","Data":"6a9fd7acf6b5c68823fb404b60df1f6b626b2337b7239e9092b34653250c4c8c"}
Oct 30 22:55:28.111792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.111659    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq" event={"ID":"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4","Type":"ContainerStarted","Data":"328a048976ca2d856ad4dada979945dbbf2ab492a519197ede23f06c5a894160"}
Oct 30 22:55:28.113122 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.113101    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-2cbd22" event={"ID":"b5271e81-ba3e-4fdd-9fbc-691894ac13bd","Type":"ContainerStarted","Data":"fae6b9e88c5a7fe17f6069f115e360a0b45e954a8d361c1a2b332ccfa61b865c"}
Oct 30 22:55:28.159951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.159844    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq" podStartSLOduration=2.15982123 podStartE2EDuration="2.15982123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:28.145592943 +0000 UTC m=+2912.184531822" watchObservedRunningTime="2026-10-30 22:55:28.15982123 +0000 UTC m=+2912.198760118"
Oct 30 22:55:28.357422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.357000    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:28.474287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.472934    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-xks-ca\") pod \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") "
Oct 30 22:55:28.474287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.473090    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-kube-api-access-vh7sn\" (UniqueName: \"kubernetes.io/projected/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-kube-api-access-vh7sn\") pod \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") "
Oct 30 22:55:28.474287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.473135    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-openshift-ca\") pod \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\" (UID: \"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d\") "
Oct 30 22:55:28.474984 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.474929    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-openshift-ca" pod "efde7b5e-b65f-450f-b47c-7e5fb76f8e1d" (UID: "efde7b5e-b65f-450f-b47c-7e5fb76f8e1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:28.477799 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.477759    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-kube-api-access-vh7sn" pod "efde7b5e-b65f-450f-b47c-7e5fb76f8e1d" (UID: "efde7b5e-b65f-450f-b47c-7e5fb76f8e1d"). InnerVolumeSpecName "kube-api-access-vh7sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:28.478038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.478012    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-xks-ca" pod "efde7b5e-b65f-450f-b47c-7e5fb76f8e1d" (UID: "efde7b5e-b65f-450f-b47c-7e5fb76f8e1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:28.575180 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.575094    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:28.575180 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.575129    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7sn\" (UniqueName: \"kubernetes.io/projected/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-kube-api-access-vh7sn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:28.575180 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:28.575144    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:29.127425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.127382    4581 generic.go:356] "Generic (PLEG): container finished" podID="ed08f5ea-cca4-4188-b81d-e2b2f8f5003b" containerID="70e286f583eef379bad5e48f26ad7ddf0efd61e707db33f83d303e386cd35f69" exitCode=0
Oct 30 22:55:29.127888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.127517    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd" event={"ID":"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b","Type":"ContainerDied","Data":"70e286f583eef379bad5e48f26ad7ddf0efd61e707db33f83d303e386cd35f69"}
Oct 30 22:55:29.130110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.130053    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78" event={"ID":"f8e2b178-91ce-4aa8-bf90-64b6066054f5","Type":"ContainerStarted","Data":"607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50"}
Oct 30 22:55:29.134260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.134166    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"
Oct 30 22:55:29.134553 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.134523    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5" event={"ID":"efde7b5e-b65f-450f-b47c-7e5fb76f8e1d","Type":"ContainerDied","Data":"90fb6e0054432c2dfcd0768a7b188d0aa0a4e36bba677df78653d231e2d60cef"}
Oct 30 22:55:29.134686 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.134571    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90fb6e0054432c2dfcd0768a7b188d0aa0a4e36bba677df78653d231e2d60cef"
Oct 30 22:55:29.143795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.143758    4581 generic.go:356] "Generic (PLEG): container finished" podID="bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4" containerID="6a9fd7acf6b5c68823fb404b60df1f6b626b2337b7239e9092b34653250c4c8c" exitCode=0
Oct 30 22:55:29.143921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.143881    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq" event={"ID":"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4","Type":"ContainerDied","Data":"6a9fd7acf6b5c68823fb404b60df1f6b626b2337b7239e9092b34653250c4c8c"}
Oct 30 22:55:29.202989 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.202930    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78" podStartSLOduration=2.202913282 podStartE2EDuration="2.202913282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:29.200993236 +0000 UTC m=+2913.239932127" watchObservedRunningTime="2026-10-30 22:55:29.202913282 +0000 UTC m=+2913.241852168"
Oct 30 22:55:29.701292 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:29.701257    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:30.155599 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.155559    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-2cbd22" event={"ID":"b5271e81-ba3e-4fdd-9fbc-691894ac13bd","Type":"ContainerStarted","Data":"e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a"}
Oct 30 22:55:30.194807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.194126    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7281-2cbd22" podStartSLOduration=1.558421857 podStartE2EDuration="4.194104167s" totalImagesPullingTime="2.63568231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:30.188166525 +0000 UTC m=+2914.227105410" watchObservedRunningTime="2026-10-30 22:55:30.194104167 +0000 UTC m=+2914.233043054"
Oct 30 22:55:30.410861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.410796    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:30.414113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.414091    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:30.608120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.608078    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-openshift-ca\") pod \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") "
Oct 30 22:55:30.608344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.608265    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-xks-ca\") pod \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") "
Oct 30 22:55:30.608344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.608311    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-openshift-ca\") pod \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") "
Oct 30 22:55:30.608460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.608443    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-kube-api-access-g2nxl\" (UniqueName: \"kubernetes.io/projected/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-kube-api-access-g2nxl\") pod \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\" (UID: \"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b\") "
Oct 30 22:55:30.608520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.608478    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-kube-api-access-zx295\" (UniqueName: \"kubernetes.io/projected/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-kube-api-access-zx295\") pod \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") "
Oct 30 22:55:30.608520 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.608482    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-openshift-ca" pod "bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4" (UID: "bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:30.608699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.608672    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-xks-ca\") pod \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\" (UID: \"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4\") "
Oct 30 22:55:30.608699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.608689    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-openshift-ca" pod "ed08f5ea-cca4-4188-b81d-e2b2f8f5003b" (UID: "ed08f5ea-cca4-4188-b81d-e2b2f8f5003b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:30.609370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.609341    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:30.609370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.609371    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:30.610903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.610871    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-xks-ca" pod "ed08f5ea-cca4-4188-b81d-e2b2f8f5003b" (UID: "ed08f5ea-cca4-4188-b81d-e2b2f8f5003b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:30.611031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.611007    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-kube-api-access-zx295" pod "bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4" (UID: "bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4"). InnerVolumeSpecName "kube-api-access-zx295". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:30.611191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.611166    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-xks-ca" pod "bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4" (UID: "bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:30.611331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.611169    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-kube-api-access-g2nxl" pod "ed08f5ea-cca4-4188-b81d-e2b2f8f5003b" (UID: "ed08f5ea-cca4-4188-b81d-e2b2f8f5003b"). InnerVolumeSpecName "kube-api-access-g2nxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:30.710607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.710517    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:30.710607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.710549    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2nxl\" (UniqueName: \"kubernetes.io/projected/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b-kube-api-access-g2nxl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:30.710607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.710563    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx295\" (UniqueName: \"kubernetes.io/projected/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-kube-api-access-zx295\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:30.710607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:30.710575    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:31.046428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.046314    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7281-2cbd22"]
Oct 30 22:55:31.051877 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.051845    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7281-2cbd22"]
Oct 30 22:55:31.168441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.168409    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"
Oct 30 22:55:31.168909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.168413    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq" event={"ID":"bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4","Type":"ContainerDied","Data":"328a048976ca2d856ad4dada979945dbbf2ab492a519197ede23f06c5a894160"}
Oct 30 22:55:31.168909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.168541    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="328a048976ca2d856ad4dada979945dbbf2ab492a519197ede23f06c5a894160"
Oct 30 22:55:31.170267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.170239    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"
Oct 30 22:55:31.170421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.170266    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd" event={"ID":"ed08f5ea-cca4-4188-b81d-e2b2f8f5003b","Type":"ContainerDied","Data":"a535d1eebec13506f517771ee26374fbda94cf8f62bfbf1f04ccfaab270f1072"}
Oct 30 22:55:31.170421 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.170315    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a535d1eebec13506f517771ee26374fbda94cf8f62bfbf1f04ccfaab270f1072"
Oct 30 22:55:31.264846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.264813    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"]
Oct 30 22:55:31.265587 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265571    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4" containerName="revoke-keys"
Oct 30 22:55:31.265639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265588    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4" containerName="revoke-keys"
Oct 30 22:55:31.265639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265600    4581 cpu_manager.go:414] "containerMap: removing container" podUID="efde7b5e-b65f-450f-b47c-7e5fb76f8e1d" containerName="revoke-keys"
Oct 30 22:55:31.265639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265605    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efde7b5e-b65f-450f-b47c-7e5fb76f8e1d" containerName="revoke-keys"
Oct 30 22:55:31.265639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265616    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ed08f5ea-cca4-4188-b81d-e2b2f8f5003b" containerName="revoke-keys"
Oct 30 22:55:31.265639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265621    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed08f5ea-cca4-4188-b81d-e2b2f8f5003b" containerName="revoke-keys"
Oct 30 22:55:31.265795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265648    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" containerName="test-curl-7281-2cbd22"
Oct 30 22:55:31.265795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265654    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" containerName="test-curl-7281-2cbd22"
Oct 30 22:55:31.265795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265750    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" containerName="test-curl-7281-2cbd22"
Oct 30 22:55:31.265795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265767    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4" containerName="revoke-keys"
Oct 30 22:55:31.265795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265775    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="efde7b5e-b65f-450f-b47c-7e5fb76f8e1d" containerName="revoke-keys"
Oct 30 22:55:31.265795 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.265784    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed08f5ea-cca4-4188-b81d-e2b2f8f5003b" containerName="revoke-keys"
Oct 30 22:55:31.269946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.269925    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.283728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.283698    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:31.283728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.283715    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:31.283728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.283716    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:55:31.289128 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.289100    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"]
Oct 30 22:55:31.292816 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.292789    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:31.319742 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.319659    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxddp\" (UniqueName: \"kubernetes.io/projected/07d3d612-28fb-49c6-bc10-d6b305f688f1-kube-api-access-fxddp\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.319911 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.319788    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.319966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.319936    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.421343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.421298    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxddp\" (UniqueName: \"kubernetes.io/projected/07d3d612-28fb-49c6-bc10-d6b305f688f1-kube-api-access-fxddp\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.421544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.421404    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.421544 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.421523    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.422344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.422269    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.424170 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.424148    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.439659 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.439631    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxddp\" (UniqueName: \"kubernetes.io/projected/07d3d612-28fb-49c6-bc10-d6b305f688f1-kube-api-access-fxddp\") pod \"maas-api-revoke-sub-0728251b-2jc45\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.584608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.584522    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:31.917444 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:31.917416    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"]
Oct 30 22:55:31.918595 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:31.918567    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d3d612_28fb_49c6_bc10_d6b305f688f1.slice/crio-4fc13e8d666031ef0479abb8fd5b1a62557afa64a7189217cf8d93694549315f WatchSource:0}: Error finding container 4fc13e8d666031ef0479abb8fd5b1a62557afa64a7189217cf8d93694549315f: Status 404 returned error can't find the container with id 4fc13e8d666031ef0479abb8fd5b1a62557afa64a7189217cf8d93694549315f
Oct 30 22:55:32.030837 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:32.030805    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"]
Oct 30 22:55:32.035650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:32.035618    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56820a1-qdbp5"]
Oct 30 22:55:32.099477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:32.099441    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8659f1-6981-4149-929b-6dc69fafc3a8" path="/var/lib/kubelet/pods/3d8659f1-6981-4149-929b-6dc69fafc3a8/volumes"
Oct 30 22:55:32.182099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:32.182063    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45" event={"ID":"07d3d612-28fb-49c6-bc10-d6b305f688f1","Type":"ContainerStarted","Data":"4fc13e8d666031ef0479abb8fd5b1a62557afa64a7189217cf8d93694549315f"}
Oct 30 22:55:32.182540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:32.182291    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7281-2cbd22" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" containerName="test-curl-7281-2cbd22" containerID="cri-o://e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a" gracePeriod=2
Oct 30 22:55:33.194571 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:33.194479    4581 generic.go:356] "Generic (PLEG): container finished" podID="07d3d612-28fb-49c6-bc10-d6b305f688f1" containerID="d01fc5dc710310fe7178d504eaacc5bcfff79a0800ab6decf3840dbdb9c7ba4c" exitCode=0
Oct 30 22:55:33.194964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:33.194573    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45" event={"ID":"07d3d612-28fb-49c6-bc10-d6b305f688f1","Type":"ContainerDied","Data":"d01fc5dc710310fe7178d504eaacc5bcfff79a0800ab6decf3840dbdb9c7ba4c"}
Oct 30 22:55:34.404344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.404314    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:34.461995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.461963    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:34.506454 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.506421    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-openshift-ca\") pod \"07d3d612-28fb-49c6-bc10-d6b305f688f1\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") "
Oct 30 22:55:34.506657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.506482    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-xks-ca\") pod \"07d3d612-28fb-49c6-bc10-d6b305f688f1\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") "
Oct 30 22:55:34.506657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.506636    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d3d612-28fb-49c6-bc10-d6b305f688f1-kube-api-access-fxddp\" (UniqueName: \"kubernetes.io/projected/07d3d612-28fb-49c6-bc10-d6b305f688f1-kube-api-access-fxddp\") pod \"07d3d612-28fb-49c6-bc10-d6b305f688f1\" (UID: \"07d3d612-28fb-49c6-bc10-d6b305f688f1\") "
Oct 30 22:55:34.506807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.506776    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-openshift-ca" pod "07d3d612-28fb-49c6-bc10-d6b305f688f1" (UID: "07d3d612-28fb-49c6-bc10-d6b305f688f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:34.507188 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.507168    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:34.508905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.508878    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-xks-ca" pod "07d3d612-28fb-49c6-bc10-d6b305f688f1" (UID: "07d3d612-28fb-49c6-bc10-d6b305f688f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:34.508905 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.508886    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d3d612-28fb-49c6-bc10-d6b305f688f1-kube-api-access-fxddp" pod "07d3d612-28fb-49c6-bc10-d6b305f688f1" (UID: "07d3d612-28fb-49c6-bc10-d6b305f688f1"). InnerVolumeSpecName "kube-api-access-fxddp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:34.607993 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.607954    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5271e81-ba3e-4fdd-9fbc-691894ac13bd-kube-api-access-jvjxj\" (UniqueName: \"kubernetes.io/projected/b5271e81-ba3e-4fdd-9fbc-691894ac13bd-kube-api-access-jvjxj\") pod \"b5271e81-ba3e-4fdd-9fbc-691894ac13bd\" (UID: \"b5271e81-ba3e-4fdd-9fbc-691894ac13bd\") "
Oct 30 22:55:34.608497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.608479    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxddp\" (UniqueName: \"kubernetes.io/projected/07d3d612-28fb-49c6-bc10-d6b305f688f1-kube-api-access-fxddp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:34.608616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.608499    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d3d612-28fb-49c6-bc10-d6b305f688f1-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:34.610363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.610333    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5271e81-ba3e-4fdd-9fbc-691894ac13bd-kube-api-access-jvjxj" pod "b5271e81-ba3e-4fdd-9fbc-691894ac13bd" (UID: "b5271e81-ba3e-4fdd-9fbc-691894ac13bd"). InnerVolumeSpecName "kube-api-access-jvjxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:34.710468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:34.710383    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjxj\" (UniqueName: \"kubernetes.io/projected/b5271e81-ba3e-4fdd-9fbc-691894ac13bd-kube-api-access-jvjxj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:35.213333 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.213295    4581 generic.go:356] "Generic (PLEG): container finished" podID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" containerID="e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a" exitCode=137
Oct 30 22:55:35.213516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.213343    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:35.213516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.213401    4581 scope.go:122] "RemoveContainer" containerID="e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a"
Oct 30 22:55:35.215561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.215532    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45" event={"ID":"07d3d612-28fb-49c6-bc10-d6b305f688f1","Type":"ContainerDied","Data":"4fc13e8d666031ef0479abb8fd5b1a62557afa64a7189217cf8d93694549315f"}
Oct 30 22:55:35.215680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.215573    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"
Oct 30 22:55:35.215680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.215587    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fc13e8d666031ef0479abb8fd5b1a62557afa64a7189217cf8d93694549315f"
Oct 30 22:55:35.227516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.227495    4581 scope.go:122] "RemoveContainer" containerID="e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a"
Oct 30 22:55:35.227808 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:55:35.227787    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a\": container with ID starting with e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a not found: ID does not exist" containerID="e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a"
Oct 30 22:55:35.227879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.227813    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a"} err="failed to get container status \"e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a\": rpc error: code = NotFound desc = could not find container \"e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a\": container with ID starting with e9ceafdc6486382c0dc9bb10fcfeb49381b54fcace1e4e1c4ce2d297c2ff646a not found: ID does not exist"
Oct 30 22:55:35.234568 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:55:35.234543    4581 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7281-2cbd22\" is forbidden: User \"system:node:ip-10-0-135-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-132.ec2.internal' and this object" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:35.258682 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:55:35.258648    4581 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7281-2cbd22\" is forbidden: User \"system:node:ip-10-0-135-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-132.ec2.internal' and this object" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:35.263503 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.263479    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"]
Oct 30 22:55:35.264200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.264186    4581 cpu_manager.go:414] "containerMap: removing container" podUID="07d3d612-28fb-49c6-bc10-d6b305f688f1" containerName="revoke-keys"
Oct 30 22:55:35.264264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.264203    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d3d612-28fb-49c6-bc10-d6b305f688f1" containerName="revoke-keys"
Oct 30 22:55:35.264375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.264366    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d3d612-28fb-49c6-bc10-d6b305f688f1" containerName="revoke-keys"
Oct 30 22:55:35.271545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.271524    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.282635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.282611    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"]
Oct 30 22:55:35.284684 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.284660    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:55:35.285428 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.285387    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:35.285541 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.285470    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:35.285762 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.285741    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:35.317252 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:55:35.317186    4581 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7281-2cbd22\" is forbidden: User \"system:node:ip-10-0-135-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-132.ec2.internal' and this object" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:35.420494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.420457    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.420494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.420495    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxf2l\" (UniqueName: \"kubernetes.io/projected/54f1aba2-1a40-496d-bf4b-1a6134294f8c-kube-api-access-cxf2l\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.420915 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.420575    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.522493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.522388    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.522788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.522766    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.522881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.522799    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxf2l\" (UniqueName: \"kubernetes.io/projected/54f1aba2-1a40-496d-bf4b-1a6134294f8c-kube-api-access-cxf2l\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.523645 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.523620    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.525134 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.525105    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.541602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.541577    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxf2l\" (UniqueName: \"kubernetes.io/projected/54f1aba2-1a40-496d-bf4b-1a6134294f8c-kube-api-access-cxf2l\") pod \"maas-api-revoke-sub-e0ad67ad-nw4lq\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.590710 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.590677    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:35.913453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:35.913425    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"]
Oct 30 22:55:35.914707 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:35.914678    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f1aba2_1a40_496d_bf4b_1a6134294f8c.slice/crio-2b6927ac6cb7ad3c126d5e20ee0fc901976b504de56e93a58ec0c18c9443920b WatchSource:0}: Error finding container 2b6927ac6cb7ad3c126d5e20ee0fc901976b504de56e93a58ec0c18c9443920b: Status 404 returned error can't find the container with id 2b6927ac6cb7ad3c126d5e20ee0fc901976b504de56e93a58ec0c18c9443920b
Oct 30 22:55:36.100975 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:36.100942    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" path="/var/lib/kubelet/pods/b5271e81-ba3e-4fdd-9fbc-691894ac13bd/volumes"
Oct 30 22:55:36.106142 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:55:36.106107    4581 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7281-2cbd22\" is forbidden: User \"system:node:ip-10-0-135-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-132.ec2.internal' and this object" podUID="b5271e81-ba3e-4fdd-9fbc-691894ac13bd" pod="opendatahub/test-curl-7281-2cbd22"
Oct 30 22:55:36.226146 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:36.226108    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq" event={"ID":"54f1aba2-1a40-496d-bf4b-1a6134294f8c","Type":"ContainerStarted","Data":"2b6927ac6cb7ad3c126d5e20ee0fc901976b504de56e93a58ec0c18c9443920b"}
Oct 30 22:55:36.919153 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:36.919116    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"]
Oct 30 22:55:36.923782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:36.923766    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:36.943258 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:36.943185    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"]
Oct 30 22:55:37.045035 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.044997    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.045207 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.045189    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46jk5\" (UniqueName: \"kubernetes.io/projected/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-kube-api-access-46jk5\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.045314 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.045301    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.146522 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.146489    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jk5\" (UniqueName: \"kubernetes.io/projected/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-kube-api-access-46jk5\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.146694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.146547    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.146694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.146679    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.147209 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.147180    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.149123 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.149099    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.165370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.165344    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jk5\" (UniqueName: \"kubernetes.io/projected/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-kube-api-access-46jk5\") pod \"maas-api-revoke-sub-330db704-tj4w2\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.237922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.237821    4581 generic.go:356] "Generic (PLEG): container finished" podID="54f1aba2-1a40-496d-bf4b-1a6134294f8c" containerID="77f550f86d3fc207e7f9671e557180e5f3fd5ad386c8496f8feaf6169c57f8f6" exitCode=0
Oct 30 22:55:37.238106 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.237932    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq" event={"ID":"54f1aba2-1a40-496d-bf4b-1a6134294f8c","Type":"ContainerDied","Data":"77f550f86d3fc207e7f9671e557180e5f3fd5ad386c8496f8feaf6169c57f8f6"}
Oct 30 22:55:37.238423 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.238404    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:37.501501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.501421    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"]
Oct 30 22:55:37.507343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.507314    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.528001 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.527940    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"]
Oct 30 22:55:37.554732 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.554620    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"]
Oct 30 22:55:37.657735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.657692    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l52c\" (UniqueName: \"kubernetes.io/projected/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-kube-api-access-4l52c\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.657959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.657890    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.658081 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.658056    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.702162 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.701282    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:37.715017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.714992    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:37.759802 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.759723    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l52c\" (UniqueName: \"kubernetes.io/projected/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-kube-api-access-4l52c\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.759976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.759845    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.759976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.759931    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.760794 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.760741    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.762514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.762490    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.777807 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.777784    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l52c\" (UniqueName: \"kubernetes.io/projected/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-kube-api-access-4l52c\") pod \"maas-api-revoke-sub-6745130f-rg6qs\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:37.823683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:37.823645    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:38.148716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.148688    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"]
Oct 30 22:55:38.150575 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:38.150545    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5c3a1a_a078_4076_bad2_7bbac28de4f8.slice/crio-f7d3d4be13eae6415cc01916db3bf41f97236fa83cfb35041a45cf7b045b1fa7 WatchSource:0}: Error finding container f7d3d4be13eae6415cc01916db3bf41f97236fa83cfb35041a45cf7b045b1fa7: Status 404 returned error can't find the container with id f7d3d4be13eae6415cc01916db3bf41f97236fa83cfb35041a45cf7b045b1fa7
Oct 30 22:55:38.247745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.247703    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs" event={"ID":"4e5c3a1a-a078-4076-bad2-7bbac28de4f8","Type":"ContainerStarted","Data":"f7d3d4be13eae6415cc01916db3bf41f97236fa83cfb35041a45cf7b045b1fa7"}
Oct 30 22:55:38.249462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.249428    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2" event={"ID":"7574ffd0-8f3a-4720-84c6-98a8dbaf7813","Type":"ContainerStarted","Data":"296eef6b7fd6244efcd1c439634f689fc2d04ba89e242a1972eaeed4af5b1048"}
Oct 30 22:55:38.249625 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.249479    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2" event={"ID":"7574ffd0-8f3a-4720-84c6-98a8dbaf7813","Type":"ContainerStarted","Data":"84807726a6d53a074c351f64113b904291914936235422633e776e41658b079f"}
Oct 30 22:55:38.262758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.262729    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 22:55:38.283289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.283174    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2" podStartSLOduration=2.283155417 podStartE2EDuration="2.283155417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:38.282806316 +0000 UTC m=+2922.321745204" watchObservedRunningTime="2026-10-30 22:55:38.283155417 +0000 UTC m=+2922.322094305"
Oct 30 22:55:38.490355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.490323    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:38.686586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.686560    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-xks-ca\") pod \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") "
Oct 30 22:55:38.686748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.686664    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-openshift-ca\") pod \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") "
Oct 30 22:55:38.686882 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.686866    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f1aba2-1a40-496d-bf4b-1a6134294f8c-kube-api-access-cxf2l\" (UniqueName: \"kubernetes.io/projected/54f1aba2-1a40-496d-bf4b-1a6134294f8c-kube-api-access-cxf2l\") pod \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\" (UID: \"54f1aba2-1a40-496d-bf4b-1a6134294f8c\") "
Oct 30 22:55:38.687091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.687059    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-openshift-ca" pod "54f1aba2-1a40-496d-bf4b-1a6134294f8c" (UID: "54f1aba2-1a40-496d-bf4b-1a6134294f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:38.687448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.687429    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:38.688865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.688835    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-xks-ca" pod "54f1aba2-1a40-496d-bf4b-1a6134294f8c" (UID: "54f1aba2-1a40-496d-bf4b-1a6134294f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:38.689438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.689414    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f1aba2-1a40-496d-bf4b-1a6134294f8c-kube-api-access-cxf2l" pod "54f1aba2-1a40-496d-bf4b-1a6134294f8c" (UID: "54f1aba2-1a40-496d-bf4b-1a6134294f8c"). InnerVolumeSpecName "kube-api-access-cxf2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:38.788921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.788875    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxf2l\" (UniqueName: \"kubernetes.io/projected/54f1aba2-1a40-496d-bf4b-1a6134294f8c-kube-api-access-cxf2l\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:38.788921 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:38.788914    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f1aba2-1a40-496d-bf4b-1a6134294f8c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:39.280721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:39.280681    4581 generic.go:356] "Generic (PLEG): container finished" podID="4e5c3a1a-a078-4076-bad2-7bbac28de4f8" containerID="03167dbbeb2cda87142ca16930a3e7ff2e59ea0c8d12d5f238a905783e226e4e" exitCode=0
Oct 30 22:55:39.281273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:39.281247    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs" event={"ID":"4e5c3a1a-a078-4076-bad2-7bbac28de4f8","Type":"ContainerDied","Data":"03167dbbeb2cda87142ca16930a3e7ff2e59ea0c8d12d5f238a905783e226e4e"}
Oct 30 22:55:39.283441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:39.283417    4581 generic.go:356] "Generic (PLEG): container finished" podID="7574ffd0-8f3a-4720-84c6-98a8dbaf7813" containerID="296eef6b7fd6244efcd1c439634f689fc2d04ba89e242a1972eaeed4af5b1048" exitCode=0
Oct 30 22:55:39.283574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:39.283504    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2" event={"ID":"7574ffd0-8f3a-4720-84c6-98a8dbaf7813","Type":"ContainerDied","Data":"296eef6b7fd6244efcd1c439634f689fc2d04ba89e242a1972eaeed4af5b1048"}
Oct 30 22:55:39.285278 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:39.285256    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"
Oct 30 22:55:39.285833 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:39.285251    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq" event={"ID":"54f1aba2-1a40-496d-bf4b-1a6134294f8c","Type":"ContainerDied","Data":"2b6927ac6cb7ad3c126d5e20ee0fc901976b504de56e93a58ec0c18c9443920b"}
Oct 30 22:55:39.285833 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:39.285387    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b6927ac6cb7ad3c126d5e20ee0fc901976b504de56e93a58ec0c18c9443920b"
Oct 30 22:55:40.534667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.534641    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:40.538274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.538255    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:40.614846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.614808    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-xks-ca\") pod \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") "
Oct 30 22:55:40.615047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.614893    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-kube-api-access-46jk5\" (UniqueName: \"kubernetes.io/projected/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-kube-api-access-46jk5\") pod \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") "
Oct 30 22:55:40.615047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.614945    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-openshift-ca\") pod \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") "
Oct 30 22:55:40.615047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.614993    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-openshift-ca\") pod \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\" (UID: \"7574ffd0-8f3a-4720-84c6-98a8dbaf7813\") "
Oct 30 22:55:40.615284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.615069    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-kube-api-access-4l52c\" (UniqueName: \"kubernetes.io/projected/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-kube-api-access-4l52c\") pod \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") "
Oct 30 22:55:40.615284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.615184    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-xks-ca\") pod \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\" (UID: \"4e5c3a1a-a078-4076-bad2-7bbac28de4f8\") "
Oct 30 22:55:40.620337 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.620301    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-openshift-ca" pod "7574ffd0-8f3a-4720-84c6-98a8dbaf7813" (UID: "7574ffd0-8f3a-4720-84c6-98a8dbaf7813"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:40.620484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.620467    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-openshift-ca" pod "4e5c3a1a-a078-4076-bad2-7bbac28de4f8" (UID: "4e5c3a1a-a078-4076-bad2-7bbac28de4f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:40.620596 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.620560    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-xks-ca" pod "4e5c3a1a-a078-4076-bad2-7bbac28de4f8" (UID: "4e5c3a1a-a078-4076-bad2-7bbac28de4f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:40.623231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.623187    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-kube-api-access-46jk5" pod "7574ffd0-8f3a-4720-84c6-98a8dbaf7813" (UID: "7574ffd0-8f3a-4720-84c6-98a8dbaf7813"). InnerVolumeSpecName "kube-api-access-46jk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:40.623330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.623181    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-xks-ca" pod "7574ffd0-8f3a-4720-84c6-98a8dbaf7813" (UID: "7574ffd0-8f3a-4720-84c6-98a8dbaf7813"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:40.624549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.624527    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-kube-api-access-4l52c" pod "4e5c3a1a-a078-4076-bad2-7bbac28de4f8" (UID: "4e5c3a1a-a078-4076-bad2-7bbac28de4f8"). InnerVolumeSpecName "kube-api-access-4l52c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:40.717205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.717174    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:40.717205 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.717203    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46jk5\" (UniqueName: \"kubernetes.io/projected/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-kube-api-access-46jk5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:40.717438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.717238    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:40.717438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.717249    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7574ffd0-8f3a-4720-84c6-98a8dbaf7813-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:40.717438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.717258    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l52c\" (UniqueName: \"kubernetes.io/projected/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-kube-api-access-4l52c\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:40.717438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:40.717268    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5c3a1a-a078-4076-bad2-7bbac28de4f8-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:40.818813 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:55:40.818776    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:55:41.306206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.306178    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"
Oct 30 22:55:41.306452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.306197    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs" event={"ID":"4e5c3a1a-a078-4076-bad2-7bbac28de4f8","Type":"ContainerDied","Data":"f7d3d4be13eae6415cc01916db3bf41f97236fa83cfb35041a45cf7b045b1fa7"}
Oct 30 22:55:41.306452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.306266    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7d3d4be13eae6415cc01916db3bf41f97236fa83cfb35041a45cf7b045b1fa7"
Oct 30 22:55:41.307752 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.307733    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"
Oct 30 22:55:41.307874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.307771    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2" event={"ID":"7574ffd0-8f3a-4720-84c6-98a8dbaf7813","Type":"ContainerDied","Data":"84807726a6d53a074c351f64113b904291914936235422633e776e41658b079f"}
Oct 30 22:55:41.307874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.307810    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84807726a6d53a074c351f64113b904291914936235422633e776e41658b079f"
Oct 30 22:55:41.693835 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.693794    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"]
Oct 30 22:55:41.694701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694680    4581 cpu_manager.go:414] "containerMap: removing container" podUID="54f1aba2-1a40-496d-bf4b-1a6134294f8c" containerName="revoke-keys"
Oct 30 22:55:41.694701 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694701    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f1aba2-1a40-496d-bf4b-1a6134294f8c" containerName="revoke-keys"
Oct 30 22:55:41.694830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694724    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4e5c3a1a-a078-4076-bad2-7bbac28de4f8" containerName="revoke-keys"
Oct 30 22:55:41.694830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694732    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5c3a1a-a078-4076-bad2-7bbac28de4f8" containerName="revoke-keys"
Oct 30 22:55:41.694830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694751    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7574ffd0-8f3a-4720-84c6-98a8dbaf7813" containerName="revoke-keys"
Oct 30 22:55:41.694830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694758    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7574ffd0-8f3a-4720-84c6-98a8dbaf7813" containerName="revoke-keys"
Oct 30 22:55:41.694976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694897    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5c3a1a-a078-4076-bad2-7bbac28de4f8" containerName="revoke-keys"
Oct 30 22:55:41.694976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694908    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7574ffd0-8f3a-4720-84c6-98a8dbaf7813" containerName="revoke-keys"
Oct 30 22:55:41.694976 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.694923    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f1aba2-1a40-496d-bf4b-1a6134294f8c" containerName="revoke-keys"
Oct 30 22:55:41.698937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.698918    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.712608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.712583    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:41.712698 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.712616    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:41.722335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.722307    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"]
Oct 30 22:55:41.729529 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.729395    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:55:41.729658 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.729559    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:41.739415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.739352    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjpc\" (UniqueName: \"kubernetes.io/projected/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-kube-api-access-hwjpc\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.739575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.739538    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.739683 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.739667    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.841257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.841193    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjpc\" (UniqueName: \"kubernetes.io/projected/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-kube-api-access-hwjpc\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.841469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.841357    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.841469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.841462    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.842142 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.842113    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.843957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.843930    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:41.858914 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:41.858885    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjpc\" (UniqueName: \"kubernetes.io/projected/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-kube-api-access-hwjpc\") pod \"maas-api-revoke-sub-24295132-qbh7m\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:42.014104 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:42.014022    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:42.336569 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:42.336547    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"]
Oct 30 22:55:42.338439 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:42.338404    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70aae6ab_0b1c_45e4_b7f9_38e86bcf142b.slice/crio-db90d61072ebc5a59a0c16eada06c28e7380e3fab71243a5fe36b66aa937d842 WatchSource:0}: Error finding container db90d61072ebc5a59a0c16eada06c28e7380e3fab71243a5fe36b66aa937d842: Status 404 returned error can't find the container with id db90d61072ebc5a59a0c16eada06c28e7380e3fab71243a5fe36b66aa937d842
Oct 30 22:55:43.032383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:43.032296    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"]
Oct 30 22:55:43.038926 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:43.038897    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e0bf3e-rt87m"]
Oct 30 22:55:43.326159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:43.326067    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m" event={"ID":"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b","Type":"ContainerStarted","Data":"18995c2b7f2dfb2f4c4b5e47110dc353bd74c2509b40cf626db8e6ff05b4a706"}
Oct 30 22:55:43.326159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:43.326122    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m" event={"ID":"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b","Type":"ContainerStarted","Data":"db90d61072ebc5a59a0c16eada06c28e7380e3fab71243a5fe36b66aa937d842"}
Oct 30 22:55:43.364565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:43.364512    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m" podStartSLOduration=2.364497269 podStartE2EDuration="2.364497269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:43.35978016 +0000 UTC m=+2927.398719046" watchObservedRunningTime="2026-10-30 22:55:43.364497269 +0000 UTC m=+2927.403436155"
Oct 30 22:55:44.103903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:44.103869    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e" path="/var/lib/kubelet/pods/b3f176d1-11ae-4a50-94fa-c5a0e6cf8a9e/volumes"
Oct 30 22:55:44.336670 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:44.336638    4581 generic.go:356] "Generic (PLEG): container finished" podID="70aae6ab-0b1c-45e4-b7f9-38e86bcf142b" containerID="18995c2b7f2dfb2f4c4b5e47110dc353bd74c2509b40cf626db8e6ff05b4a706" exitCode=0
Oct 30 22:55:44.336842 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:44.336726    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m" event={"ID":"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b","Type":"ContainerDied","Data":"18995c2b7f2dfb2f4c4b5e47110dc353bd74c2509b40cf626db8e6ff05b4a706"}
Oct 30 22:55:45.557724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.557688    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:45.614542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.614514    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-openshift-ca\") pod \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") "
Oct 30 22:55:45.614704 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.614679    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-xks-ca\") pod \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") "
Oct 30 22:55:45.614813 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.614796    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-kube-api-access-hwjpc\" (UniqueName: \"kubernetes.io/projected/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-kube-api-access-hwjpc\") pod \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\" (UID: \"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b\") "
Oct 30 22:55:45.614942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.614920    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-openshift-ca" pod "70aae6ab-0b1c-45e4-b7f9-38e86bcf142b" (UID: "70aae6ab-0b1c-45e4-b7f9-38e86bcf142b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:45.615776 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.615749    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:45.617131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.617106    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-xks-ca" pod "70aae6ab-0b1c-45e4-b7f9-38e86bcf142b" (UID: "70aae6ab-0b1c-45e4-b7f9-38e86bcf142b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:45.617299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.617210    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-kube-api-access-hwjpc" pod "70aae6ab-0b1c-45e4-b7f9-38e86bcf142b" (UID: "70aae6ab-0b1c-45e4-b7f9-38e86bcf142b"). InnerVolumeSpecName "kube-api-access-hwjpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:45.717154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.717124    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjpc\" (UniqueName: \"kubernetes.io/projected/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-kube-api-access-hwjpc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:45.717154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:45.717150    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:46.359319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:46.359211    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"
Oct 30 22:55:46.359319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:46.359236    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m" event={"ID":"70aae6ab-0b1c-45e4-b7f9-38e86bcf142b","Type":"ContainerDied","Data":"db90d61072ebc5a59a0c16eada06c28e7380e3fab71243a5fe36b66aa937d842"}
Oct 30 22:55:46.359524 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:46.359324    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db90d61072ebc5a59a0c16eada06c28e7380e3fab71243a5fe36b66aa937d842"
Oct 30 22:55:47.825078 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.825038    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"]
Oct 30 22:55:47.825773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.825758    4581 cpu_manager.go:414] "containerMap: removing container" podUID="70aae6ab-0b1c-45e4-b7f9-38e86bcf142b" containerName="revoke-keys"
Oct 30 22:55:47.825829 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.825775    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70aae6ab-0b1c-45e4-b7f9-38e86bcf142b" containerName="revoke-keys"
Oct 30 22:55:47.825940 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.825930    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="70aae6ab-0b1c-45e4-b7f9-38e86bcf142b" containerName="revoke-keys"
Oct 30 22:55:47.830044 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.830026    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:47.850573 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.850541    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"]
Oct 30 22:55:47.854379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.854350    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:47.854379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.854382    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:47.854583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.854388    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:47.854583 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.854455    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:55:47.950957 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.950923    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:47.951160 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.951018    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:47.951275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:47.951212    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwg5h\" (UniqueName: \"kubernetes.io/projected/9dbb8c63-5389-46c3-8d69-46880b94300e-kube-api-access-cwg5h\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:48.052970 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:48.052936    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:48.053191 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:48.053046    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:48.053309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:48.053290    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg5h\" (UniqueName: \"kubernetes.io/projected/9dbb8c63-5389-46c3-8d69-46880b94300e-kube-api-access-cwg5h\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:48.053780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:48.053745    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:48.055758 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:48.055729    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:48.070865 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:48.070815    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwg5h\" (UniqueName: \"kubernetes.io/projected/9dbb8c63-5389-46c3-8d69-46880b94300e-kube-api-access-cwg5h\") pod \"maas-api-revoke-sub-f88d9b76-vpwkr\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:48.145437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:48.145408    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:48.467131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:48.467099    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"]
Oct 30 22:55:48.468151 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:48.468122    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbb8c63_5389_46c3_8d69_46880b94300e.slice/crio-48a424111c13d92ee3dbc42b0cbfb841d90f77eb84cf9f00be32881d1cb7997f WatchSource:0}: Error finding container 48a424111c13d92ee3dbc42b0cbfb841d90f77eb84cf9f00be32881d1cb7997f: Status 404 returned error can't find the container with id 48a424111c13d92ee3dbc42b0cbfb841d90f77eb84cf9f00be32881d1cb7997f
Oct 30 22:55:49.391022 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:49.390994    4581 generic.go:356] "Generic (PLEG): container finished" podID="9dbb8c63-5389-46c3-8d69-46880b94300e" containerID="e5643a05b7e1daae68a6fb18d460733d74e6db3020d218d604dd437e78351f82" exitCode=0
Oct 30 22:55:49.391487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:49.391084    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr" event={"ID":"9dbb8c63-5389-46c3-8d69-46880b94300e","Type":"ContainerDied","Data":"e5643a05b7e1daae68a6fb18d460733d74e6db3020d218d604dd437e78351f82"}
Oct 30 22:55:49.391487 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:49.391186    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr" event={"ID":"9dbb8c63-5389-46c3-8d69-46880b94300e","Type":"ContainerStarted","Data":"48a424111c13d92ee3dbc42b0cbfb841d90f77eb84cf9f00be32881d1cb7997f"}
Oct 30 22:55:50.582300 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.582264    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:50.694502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.694414    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-xks-ca\") pod \"9dbb8c63-5389-46c3-8d69-46880b94300e\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") "
Oct 30 22:55:50.694661 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.694598    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbb8c63-5389-46c3-8d69-46880b94300e-kube-api-access-cwg5h\" (UniqueName: \"kubernetes.io/projected/9dbb8c63-5389-46c3-8d69-46880b94300e-kube-api-access-cwg5h\") pod \"9dbb8c63-5389-46c3-8d69-46880b94300e\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") "
Oct 30 22:55:50.694745 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.694730    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-openshift-ca\") pod \"9dbb8c63-5389-46c3-8d69-46880b94300e\" (UID: \"9dbb8c63-5389-46c3-8d69-46880b94300e\") "
Oct 30 22:55:50.695151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.695117    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-openshift-ca" pod "9dbb8c63-5389-46c3-8d69-46880b94300e" (UID: "9dbb8c63-5389-46c3-8d69-46880b94300e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:50.695700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.695675    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:50.696860 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.696823    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-xks-ca" pod "9dbb8c63-5389-46c3-8d69-46880b94300e" (UID: "9dbb8c63-5389-46c3-8d69-46880b94300e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:50.697017 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.696981    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbb8c63-5389-46c3-8d69-46880b94300e-kube-api-access-cwg5h" pod "9dbb8c63-5389-46c3-8d69-46880b94300e" (UID: "9dbb8c63-5389-46c3-8d69-46880b94300e"). InnerVolumeSpecName "kube-api-access-cwg5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:50.797232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.797181    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwg5h\" (UniqueName: \"kubernetes.io/projected/9dbb8c63-5389-46c3-8d69-46880b94300e-kube-api-access-cwg5h\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:50.797232 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:50.797211    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbb8c63-5389-46c3-8d69-46880b94300e-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:55:51.411411 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:51.411379    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"
Oct 30 22:55:51.411604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:51.411420    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr" event={"ID":"9dbb8c63-5389-46c3-8d69-46880b94300e","Type":"ContainerDied","Data":"48a424111c13d92ee3dbc42b0cbfb841d90f77eb84cf9f00be32881d1cb7997f"}
Oct 30 22:55:51.411604 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:51.411471    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a424111c13d92ee3dbc42b0cbfb841d90f77eb84cf9f00be32881d1cb7997f"
Oct 30 22:55:58.340853 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.340817    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"]
Oct 30 22:55:58.341657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.341640    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9dbb8c63-5389-46c3-8d69-46880b94300e" containerName="revoke-keys"
Oct 30 22:55:58.341718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.341659    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbb8c63-5389-46c3-8d69-46880b94300e" containerName="revoke-keys"
Oct 30 22:55:58.341809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.341800    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbb8c63-5389-46c3-8d69-46880b94300e" containerName="revoke-keys"
Oct 30 22:55:58.345540 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.345520    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.359960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.359916    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:58.360139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.360005    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:58.360139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.360071    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:58.360139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.360120    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:55:58.367919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.367886    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"]
Oct 30 22:55:58.419101 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.419070    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwhw\" (UniqueName: \"kubernetes.io/projected/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-kube-api-access-dhwhw\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.419307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.419242    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.419460 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.419441    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.521185 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.521150    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.521427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.521275    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwhw\" (UniqueName: \"kubernetes.io/projected/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-kube-api-access-dhwhw\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.521427 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.521400    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.522097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.522066    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.523818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.523782    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.539347 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.539320    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwhw\" (UniqueName: \"kubernetes.io/projected/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-kube-api-access-dhwhw\") pod \"maas-api-revoke-sub-0741c530-zdc5m\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.662934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.662901    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:55:58.979267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:58.979232    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"]
Oct 30 22:55:58.980882 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:55:58.980855    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b46a98a_c1b7_40e2_88ab_b0e27b59bf6e.slice/crio-b70ec38715b9b9574da288a4f9856d23533c0d030790581e72ce6e4cfc94dc28 WatchSource:0}: Error finding container b70ec38715b9b9574da288a4f9856d23533c0d030790581e72ce6e4cfc94dc28: Status 404 returned error can't find the container with id b70ec38715b9b9574da288a4f9856d23533c0d030790581e72ce6e4cfc94dc28
Oct 30 22:55:59.504151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:59.504112    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m" event={"ID":"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e","Type":"ContainerStarted","Data":"fea969036d9d607c39dd602ef6b3053137670b2cb455a346717dc5e517f52f0a"}
Oct 30 22:55:59.504657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:59.504176    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m" event={"ID":"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e","Type":"ContainerStarted","Data":"b70ec38715b9b9574da288a4f9856d23533c0d030790581e72ce6e4cfc94dc28"}
Oct 30 22:55:59.543256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:55:59.543180    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m" podStartSLOduration=1.543166114 podStartE2EDuration="1.543166114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:59.540129546 +0000 UTC m=+2943.579068433" watchObservedRunningTime="2026-10-30 22:55:59.543166114 +0000 UTC m=+2943.582104999"
Oct 30 22:56:00.514730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:00.514695    4581 generic.go:356] "Generic (PLEG): container finished" podID="6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e" containerID="fea969036d9d607c39dd602ef6b3053137670b2cb455a346717dc5e517f52f0a" exitCode=0
Oct 30 22:56:00.515261 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:00.514771    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m" event={"ID":"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e","Type":"ContainerDied","Data":"fea969036d9d607c39dd602ef6b3053137670b2cb455a346717dc5e517f52f0a"}
Oct 30 22:56:01.724376 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.724348    4581 scope.go:122] "RemoveContainer" containerID="e53a8964f14588a3e0fe5f262da134b9ade8ff32728ea007435c1b261ad23edc"
Oct 30 22:56:01.737699 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.737678    4581 scope.go:122] "RemoveContainer" containerID="cf04d98797edc7ef2ee4f10634c3a89b194ae3989f924708ae94cafd7ffa9e41"
Oct 30 22:56:01.739094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.739071    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:56:01.750819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.750794    4581 scope.go:122] "RemoveContainer" containerID="6a40ed284bfc79983f7a5db1808d89c70a2c304cc31815cae603c0ad6c2b09c4"
Oct 30 22:56:01.869621 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.869586    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-xks-ca\") pod \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") "
Oct 30 22:56:01.869809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.869671    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-openshift-ca\") pod \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") "
Oct 30 22:56:01.870013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.869993    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-kube-api-access-dhwhw\" (UniqueName: \"kubernetes.io/projected/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-kube-api-access-dhwhw\") pod \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\" (UID: \"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e\") "
Oct 30 22:56:01.870136 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.869996    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-openshift-ca" pod "6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e" (UID: "6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:01.870622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.870601    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:01.871996 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.871969    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-xks-ca" pod "6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e" (UID: "6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:01.872089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.872058    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-kube-api-access-dhwhw" pod "6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e" (UID: "6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e"). InnerVolumeSpecName "kube-api-access-dhwhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:01.971979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.971940    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwhw\" (UniqueName: \"kubernetes.io/projected/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-kube-api-access-dhwhw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:01.971979 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:01.971980    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:02.536497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:02.536401    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"
Oct 30 22:56:02.536497 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:02.536422    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m" event={"ID":"6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e","Type":"ContainerDied","Data":"b70ec38715b9b9574da288a4f9856d23533c0d030790581e72ce6e4cfc94dc28"}
Oct 30 22:56:02.536852 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:02.536524    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b70ec38715b9b9574da288a4f9856d23533c0d030790581e72ce6e4cfc94dc28"
Oct 30 22:56:09.456478 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.456444    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"]
Oct 30 22:56:09.457449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.457423    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e" containerName="revoke-keys"
Oct 30 22:56:09.457449 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.457446    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e" containerName="revoke-keys"
Oct 30 22:56:09.457692 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.457677    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e" containerName="revoke-keys"
Oct 30 22:56:09.465425 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.465403    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.489654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.489618    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0a807-openshift-default-dockercfg-vnlfm\""
Oct 30 22:56:09.499880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.499830    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"]
Oct 30 22:56:09.622320 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622278    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjcn\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-kube-api-access-jfjcn\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.622481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622352    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-podinfo\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.622543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622489    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-token\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.622543 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622515    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-envoy\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.622715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622693    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-data\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.622756 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622743    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd29e823-643c-4ef9-b94f-dbffd2461749-istiod-ca-cert\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.622850 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622833    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-credential-socket\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.622887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622865    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-socket\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.622966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.622955    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-certs\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735062    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-token\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735114    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-envoy\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735258    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-data\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735303 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735301    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd29e823-643c-4ef9-b94f-dbffd2461749-istiod-ca-cert\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735364    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-credential-socket\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735398    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-socket\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735466    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-certs\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735508    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjcn\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-kube-api-access-jfjcn\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.735671 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.735559    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-podinfo\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.739238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.737969    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd29e823-643c-4ef9-b94f-dbffd2461749-istiod-ca-cert\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.767317 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.752361    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-socket\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.767317 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.752634    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-credential-socket\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.767317 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.752866    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-certs\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.792250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.782370    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-data\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.804242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.788638    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-token\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.804242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.800210    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-podinfo\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.804242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.802059    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-envoy\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:09.817239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:09.816860    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjcn\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-kube-api-access-jfjcn\") pod \"e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") " pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:10.085284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:10.085154    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:10.424974 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:10.424087    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"]
Oct 30 22:56:10.430054 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:56:10.429993    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd29e823_643c_4ef9_b94f_dbffd2461749.slice/crio-d46665546ef2b1a658fa7c36a63e92b0d4448b6df67ff47a2732586ab4664d52 WatchSource:0}: Error finding container d46665546ef2b1a658fa7c36a63e92b0d4448b6df67ff47a2732586ab4664d52: Status 404 returned error can't find the container with id d46665546ef2b1a658fa7c36a63e92b0d4448b6df67ff47a2732586ab4664d52
Oct 30 22:56:10.433879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:10.433841    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:56:10.433995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:10.433970    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:56:10.434083 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:10.434023    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:56:10.740202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:10.740107    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" event={"ID":"dd29e823-643c-4ef9-b94f-dbffd2461749","Type":"ContainerStarted","Data":"1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2"}
Oct 30 22:56:10.740202 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:10.740179    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" event={"ID":"dd29e823-643c-4ef9-b94f-dbffd2461749","Type":"ContainerStarted","Data":"d46665546ef2b1a658fa7c36a63e92b0d4448b6df67ff47a2732586ab4664d52"}
Oct 30 22:56:10.785491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:10.785423    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podStartSLOduration=1.785403018 podStartE2EDuration="1.785403018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:10.778613363 +0000 UTC m=+2954.817552250" watchObservedRunningTime="2026-10-30 22:56:10.785403018 +0000 UTC m=+2954.824341904"
Oct 30 22:56:10.820793 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:56:10.820745    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:56:11.085904 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:11.085795    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:11.094036 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:11.094008    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:11.753728 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:11.753692    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:11.757168 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:11.757142    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:56:15.036200 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.036168    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"]
Oct 30 22:56:15.041560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.041531    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:15.156243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.151704    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daea72b-b631-4216-a051-8155f0c0f1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws\" (UID: \"2daea72b-b631-4216-a051-8155f0c0f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:15.156243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.151803    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkk9\" (UniqueName: \"kubernetes.io/projected/2daea72b-b631-4216-a051-8155f0c0f1c1-kube-api-access-bnkk9\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws\" (UID: \"2daea72b-b631-4216-a051-8155f0c0f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:15.187244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.184187    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0a807\""
Oct 30 22:56:15.187244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.184352    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0a807-dockercfg-rxskz\""
Oct 30 22:56:15.323598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.318846    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daea72b-b631-4216-a051-8155f0c0f1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws\" (UID: \"2daea72b-b631-4216-a051-8155f0c0f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:15.352250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.351705    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkk9\" (UniqueName: \"kubernetes.io/projected/2daea72b-b631-4216-a051-8155f0c0f1c1-kube-api-access-bnkk9\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws\" (UID: \"2daea72b-b631-4216-a051-8155f0c0f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:15.352250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.329543    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"]
Oct 30 22:56:15.352250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.321037    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daea72b-b631-4216-a051-8155f0c0f1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws\" (UID: \"2daea72b-b631-4216-a051-8155f0c0f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:15.391078 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.390980    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkk9\" (UniqueName: \"kubernetes.io/projected/2daea72b-b631-4216-a051-8155f0c0f1c1-kube-api-access-bnkk9\") pod \"payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws\" (UID: \"2daea72b-b631-4216-a051-8155f0c0f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:15.661635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:15.661605    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:16.001157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:16.001059    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"]
Oct 30 22:56:16.010741 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:56:16.010706    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2daea72b_b631_4216_a051_8155f0c0f1c1.slice/crio-3b30901799495161fcf3814349ba083a21b3ff24e7bd1cf3937085f4e626ce0c WatchSource:0}: Error finding container 3b30901799495161fcf3814349ba083a21b3ff24e7bd1cf3937085f4e626ce0c: Status 404 returned error can't find the container with id 3b30901799495161fcf3814349ba083a21b3ff24e7bd1cf3937085f4e626ce0c
Oct 30 22:56:16.823858 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:16.823817    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws" event={"ID":"2daea72b-b631-4216-a051-8155f0c0f1c1","Type":"ContainerStarted","Data":"0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6"}
Oct 30 22:56:16.824283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:16.823890    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:16.824283 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:16.823914    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws" event={"ID":"2daea72b-b631-4216-a051-8155f0c0f1c1","Type":"ContainerStarted","Data":"3b30901799495161fcf3814349ba083a21b3ff24e7bd1cf3937085f4e626ce0c"}
Oct 30 22:56:16.863053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:16.862996    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws" podStartSLOduration=1.862979439 podStartE2EDuration="1.862979439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:16.857523514 +0000 UTC m=+2960.896462401" watchObservedRunningTime="2026-10-30 22:56:16.862979439 +0000 UTC m=+2960.901918325"
Oct 30 22:56:17.329039 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.329005    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"]
Oct 30 22:56:17.333720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.333699    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.347192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.347172    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:17.347383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.347366    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:56:17.347506 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.347409    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:56:17.347570 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.347516    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:56:17.358180 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.358153    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"]
Oct 30 22:56:17.410546 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.410513    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.410730 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.410707    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsv4q\" (UniqueName: \"kubernetes.io/projected/2310d7e0-398e-4859-a717-fd50c31ce239-kube-api-access-dsv4q\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.410831 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.410813    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.512354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.512311    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.512586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.512522    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsv4q\" (UniqueName: \"kubernetes.io/projected/2310d7e0-398e-4859-a717-fd50c31ce239-kube-api-access-dsv4q\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.512724 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.512618    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.513639 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.513608    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.516467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.516439    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.533712 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.533669    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsv4q\" (UniqueName: \"kubernetes.io/projected/2310d7e0-398e-4859-a717-fd50c31ce239-kube-api-access-dsv4q\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:17.647931 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:17.647894    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:18.071103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:18.071069    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"]
Oct 30 22:56:18.072361 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:56:18.072337    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2310d7e0_398e_4859_a717_fd50c31ce239.slice/crio-6c7e266ad5fcf65ca94b7089ba996a8452a0c5a444b0c2fd20c34ea26beb155d WatchSource:0}: Error finding container 6c7e266ad5fcf65ca94b7089ba996a8452a0c5a444b0c2fd20c34ea26beb155d: Status 404 returned error can't find the container with id 6c7e266ad5fcf65ca94b7089ba996a8452a0c5a444b0c2fd20c34ea26beb155d
Oct 30 22:56:18.849909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:18.849811    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5" event={"ID":"2310d7e0-398e-4859-a717-fd50c31ce239","Type":"ContainerStarted","Data":"fe18f5f560eebd7feda298ddd71cf603babc554c76c42dac0a2181db586c15c7"}
Oct 30 22:56:18.849909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:18.849864    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5" event={"ID":"2310d7e0-398e-4859-a717-fd50c31ce239","Type":"ContainerStarted","Data":"6c7e266ad5fcf65ca94b7089ba996a8452a0c5a444b0c2fd20c34ea26beb155d"}
Oct 30 22:56:18.890664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:18.890603    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5" podStartSLOduration=1.89058497 podStartE2EDuration="1.89058497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:18.882947874 +0000 UTC m=+2962.921886761" watchObservedRunningTime="2026-10-30 22:56:18.89058497 +0000 UTC m=+2962.929523856"
Oct 30 22:56:19.866374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:19.866331    4581 generic.go:356] "Generic (PLEG): container finished" podID="2310d7e0-398e-4859-a717-fd50c31ce239" containerID="fe18f5f560eebd7feda298ddd71cf603babc554c76c42dac0a2181db586c15c7" exitCode=0
Oct 30 22:56:19.866887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:19.866412    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5" event={"ID":"2310d7e0-398e-4859-a717-fd50c31ce239","Type":"ContainerDied","Data":"fe18f5f560eebd7feda298ddd71cf603babc554c76c42dac0a2181db586c15c7"}
Oct 30 22:56:21.101301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.101274    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:21.185045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.184948    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-openshift-ca\") pod \"2310d7e0-398e-4859-a717-fd50c31ce239\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") "
Oct 30 22:56:21.185233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.185042    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-xks-ca\") pod \"2310d7e0-398e-4859-a717-fd50c31ce239\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") "
Oct 30 22:56:21.185233 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.185145    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2310d7e0-398e-4859-a717-fd50c31ce239-kube-api-access-dsv4q\" (UniqueName: \"kubernetes.io/projected/2310d7e0-398e-4859-a717-fd50c31ce239-kube-api-access-dsv4q\") pod \"2310d7e0-398e-4859-a717-fd50c31ce239\" (UID: \"2310d7e0-398e-4859-a717-fd50c31ce239\") "
Oct 30 22:56:21.185430 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.185405    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-openshift-ca" pod "2310d7e0-398e-4859-a717-fd50c31ce239" (UID: "2310d7e0-398e-4859-a717-fd50c31ce239"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:21.186025 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.185798    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.187959 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.187922    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-xks-ca" pod "2310d7e0-398e-4859-a717-fd50c31ce239" (UID: "2310d7e0-398e-4859-a717-fd50c31ce239"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:21.188464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.188437    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2310d7e0-398e-4859-a717-fd50c31ce239-kube-api-access-dsv4q" pod "2310d7e0-398e-4859-a717-fd50c31ce239" (UID: "2310d7e0-398e-4859-a717-fd50c31ce239"). InnerVolumeSpecName "kube-api-access-dsv4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:21.287757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.287723    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2310d7e0-398e-4859-a717-fd50c31ce239-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.287757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.287751    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsv4q\" (UniqueName: \"kubernetes.io/projected/2310d7e0-398e-4859-a717-fd50c31ce239-kube-api-access-dsv4q\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:21.898066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.898034    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"
Oct 30 22:56:21.898301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.898062    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5" event={"ID":"2310d7e0-398e-4859-a717-fd50c31ce239","Type":"ContainerDied","Data":"6c7e266ad5fcf65ca94b7089ba996a8452a0c5a444b0c2fd20c34ea26beb155d"}
Oct 30 22:56:21.898301 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:21.898119    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c7e266ad5fcf65ca94b7089ba996a8452a0c5a444b0c2fd20c34ea26beb155d"
Oct 30 22:56:27.845913 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:27.845884    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:56:37.005161 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:37.005123    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"]
Oct 30 22:56:37.005642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:37.005393    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy" containerID="cri-o://49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7" gracePeriod=30
Oct 30 22:56:40.821188 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:56:40.821143    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:56:42.307324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.307296    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:56:42.345466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.345431    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e171f96-3c69-4a25-ab53-f9112d808eb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e171f96-3c69-4a25-ab53-f9112d808eb2-istiod-ca-cert\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.345668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.345535    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-credential-socket\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.345668 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.345599    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-socket\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.345785 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.345742    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-podinfo\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.345892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.345861    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-credential-socket" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.345971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.345893    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-socket" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.345971 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.345900    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e171f96-3c69-4a25-ab53-f9112d808eb2-istiod-ca-cert" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:42.346181 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.346165    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-kube-api-access-w6bp5\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-kube-api-access-w6bp5\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.346284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.346267    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-data\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.346346 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.346326    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-envoy\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.346410 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.346361    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-token\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-token\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.346560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.346531    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-data" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.346560 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.346541    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-certs\") pod \"6e171f96-3c69-4a25-ab53-f9112d808eb2\" (UID: \"6e171f96-3c69-4a25-ab53-f9112d808eb2\") "
Oct 30 22:56:42.346780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.346760    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-certs" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.347534 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.347507    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.347657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.347538    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.347657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.347557    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.347657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.347573    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.347657 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.347589    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e171f96-3c69-4a25-ab53-f9112d808eb2-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.348299 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.348277    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-podinfo" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:56:42.348746 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.348726    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-token" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:42.348951 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.348928    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-envoy" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:42.349052 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.348974    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-kube-api-access-w6bp5" pod "6e171f96-3c69-4a25-ab53-f9112d808eb2" (UID: "6e171f96-3c69-4a25-ab53-f9112d808eb2"). InnerVolumeSpecName "kube-api-access-w6bp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:42.448797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.448751    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.448797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.448789    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bp5\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-kube-api-access-w6bp5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.448797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.448804    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:42.449053 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:42.448817    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e171f96-3c69-4a25-ab53-f9112d808eb2-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:56:43.138249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.134894    4581 generic.go:356] "Generic (PLEG): container finished" podID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerID="49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7" exitCode=0
Oct 30 22:56:43.138249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.134958    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"
Oct 30 22:56:43.138249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.135242    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" event={"ID":"6e171f96-3c69-4a25-ab53-f9112d808eb2","Type":"ContainerDied","Data":"49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7"}
Oct 30 22:56:43.138249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.135288    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg" event={"ID":"6e171f96-3c69-4a25-ab53-f9112d808eb2","Type":"ContainerDied","Data":"fe6d00a68b9c70343b89dffab9dbe73fc3571fe130358d14b1b4fcaea86d64b1"}
Oct 30 22:56:43.138249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.135314    4581 scope.go:122] "RemoveContainer" containerID="49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7"
Oct 30 22:56:43.158361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.158315    4581 scope.go:122] "RemoveContainer" containerID="49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7"
Oct 30 22:56:43.158828 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:56:43.158746    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7\": container with ID starting with 49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7 not found: ID does not exist" containerID="49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7"
Oct 30 22:56:43.158828 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.158818    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7"} err="failed to get container status \"49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7\": rpc error: code = NotFound desc = could not find container \"49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7\": container with ID starting with 49d6aaaf7dad565b604da8f7bf3ad627a0534c11b042d38a31ea848731a03ff7 not found: ID does not exist"
Oct 30 22:56:43.196654 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.196620    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"]
Oct 30 22:56:43.204274 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:43.204247    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0ff5b0-openshift-default-ff46db8d5-tz9sg"]
Oct 30 22:56:44.100867 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:44.100837    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" path="/var/lib/kubelet/pods/6e171f96-3c69-4a25-ab53-f9112d808eb2/volumes"
Oct 30 22:56:54.584137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.584098    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"]
Oct 30 22:56:54.585033 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.585009    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2310d7e0-398e-4859-a717-fd50c31ce239" containerName="revoke-keys"
Oct 30 22:56:54.585156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.585034    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2310d7e0-398e-4859-a717-fd50c31ce239" containerName="revoke-keys"
Oct 30 22:56:54.585156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.585107    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy"
Oct 30 22:56:54.585156 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.585117    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy"
Oct 30 22:56:54.585359 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.585339    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e171f96-3c69-4a25-ab53-f9112d808eb2" containerName="istio-proxy"
Oct 30 22:56:54.585415 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.585361    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2310d7e0-398e-4859-a717-fd50c31ce239" containerName="revoke-keys"
Oct 30 22:56:54.589265 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.589242    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.604029 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.604006    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-80ecb3-openshift-default-dockercfg-g6zvp\""
Oct 30 22:56:54.619675 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.619643    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"]
Oct 30 22:56:54.664452 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664417    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-podinfo\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.664655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664482    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-envoy\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.664655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664580    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-certs\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.664655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664612    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-token\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.664655 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664641    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istiod-ca-cert\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.664896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664686    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-data\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.664896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664744    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d57m\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-kube-api-access-5d57m\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.664896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664860    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.664896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.664889    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-credential-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.766363 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766324    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-certs\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.766581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766391    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-token\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.766581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766427    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istiod-ca-cert\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.766581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766451    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-data\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.766581 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766537    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d57m\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-kube-api-access-5d57m\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.766812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766656    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.766812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766687    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-credential-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.766812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766751    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-podinfo\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.767032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766830    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-envoy\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.767032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.766883    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-certs\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.767032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.767008    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-data\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.767190 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.767159    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-credential-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.767419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.767393    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-socket\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.767635 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.767603    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istiod-ca-cert\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.769735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.769707    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-envoy\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.770021 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.770006    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-podinfo\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.784343 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.784308    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d57m\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-kube-api-access-5d57m\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.784642 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.784621    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-token\") pod \"e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") " pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:54.904304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:54.904272    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:55.240669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:55.240641    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"]
Oct 30 22:56:55.241562 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:56:55.241539    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf36b5e_49ce_4a82_bad4_fdfcdca3c397.slice/crio-922c5acbd15f1b75342fa66623d95dc189a84bc183b716b5dde43583ab7cab88 WatchSource:0}: Error finding container 922c5acbd15f1b75342fa66623d95dc189a84bc183b716b5dde43583ab7cab88: Status 404 returned error can't find the container with id 922c5acbd15f1b75342fa66623d95dc189a84bc183b716b5dde43583ab7cab88
Oct 30 22:56:55.244980 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:55.244940    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:56:55.245100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:55.245032    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:56:55.245100 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:55.245083    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:56:55.267307 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:55.267269    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" event={"ID":"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397","Type":"ContainerStarted","Data":"922c5acbd15f1b75342fa66623d95dc189a84bc183b716b5dde43583ab7cab88"}
Oct 30 22:56:56.292289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:56.292249    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" event={"ID":"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397","Type":"ContainerStarted","Data":"40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673"}
Oct 30 22:56:56.344574 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:56.344511    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" podStartSLOduration=2.344492114 podStartE2EDuration="2.344492114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:56.342706088 +0000 UTC m=+3000.381644976" watchObservedRunningTime="2026-10-30 22:56:56.344492114 +0000 UTC m=+3000.383430999"
Oct 30 22:56:56.905198 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:56.905159    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:56:56.906717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:56.906685    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 22:56:56.906851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:56.906747    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 22:56:57.414271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:57.414208    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:56:57.417873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:57.417846    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 22:56:57.905567 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:57.905538    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 22:56:57.905820 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:57.905797    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 22:56:58.905796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:58.905012    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 22:56:58.905796 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:58.905114    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 22:56:59.905249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:59.905177    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 22:56:59.905472 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:56:59.905295    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 22:57:00.908248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:00.904983    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 30 22:57:00.908248 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:00.905062    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 30 22:57:01.817013 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:01.816977    4581 scope.go:122] "RemoveContainer" containerID="4c2f9f17f3b525ddfdf96551901fc4001ce69e5f907422d24eb778b9709dfaa2"
Oct 30 22:57:01.835310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:01.835281    4581 scope.go:122] "RemoveContainer" containerID="40e2cbddc3036a89d2892b78c35f0f808d853be2104387eb91b91cf865d2a2cc"
Oct 30 22:57:01.930316 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:01.930279    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:57:01.931469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:01.931441    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:57:01.932077 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:01.932047    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 22:57:03.012286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:03.012249    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"]
Oct 30 22:57:03.025608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:03.025566    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c511a3-qvkd5"]
Oct 30 22:57:04.100365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:04.100327    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2310d7e0-398e-4859-a717-fd50c31ce239" path="/var/lib/kubelet/pods/2310d7e0-398e-4859-a717-fd50c31ce239/volumes"
Oct 30 22:57:10.827442 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:57:10.827358    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:57:11.434279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.434242    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"]
Oct 30 22:57:11.440962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.440940    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.480516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.480484    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"]
Oct 30 22:57:11.483607 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.483583    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:11.483775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.483660    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:11.490861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.490743    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:57:11.490861 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.490762    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:57:11.542266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.542232    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxk4\" (UniqueName: \"kubernetes.io/projected/5bd7e4d2-5b4e-415a-af27-b98a688d1724-kube-api-access-bpxk4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.542456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.542280    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.542456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.542369    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.643735 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.643689    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxk4\" (UniqueName: \"kubernetes.io/projected/5bd7e4d2-5b4e-415a-af27-b98a688d1724-kube-api-access-bpxk4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.643923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.643751    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.643923 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.643784    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.644755 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.644715    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.646716 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.646688    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.661801 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.661778    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxk4\" (UniqueName: \"kubernetes.io/projected/5bd7e4d2-5b4e-415a-af27-b98a688d1724-kube-api-access-bpxk4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:11.756510 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:11.756423    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:12.072528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:12.072501    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"]
Oct 30 22:57:12.075327 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:57:12.075293    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd7e4d2_5b4e_415a_af27_b98a688d1724.slice/crio-7b7c91d4757866f9bc914584b363c03b5b913327402ac9c076ad4d32777b5313 WatchSource:0}: Error finding container 7b7c91d4757866f9bc914584b363c03b5b913327402ac9c076ad4d32777b5313: Status 404 returned error can't find the container with id 7b7c91d4757866f9bc914584b363c03b5b913327402ac9c076ad4d32777b5313
Oct 30 22:57:12.539769 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:12.539726    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8" event={"ID":"5bd7e4d2-5b4e-415a-af27-b98a688d1724","Type":"ContainerStarted","Data":"56524657ee5a4ca2f10562b72842a9964fef3b78d12b7196c8da49d71d3b50f2"}
Oct 30 22:57:12.539950 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:12.539827    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8" event={"ID":"5bd7e4d2-5b4e-415a-af27-b98a688d1724","Type":"ContainerStarted","Data":"7b7c91d4757866f9bc914584b363c03b5b913327402ac9c076ad4d32777b5313"}
Oct 30 22:57:12.575909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:12.575851    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8" podStartSLOduration=1.575837004 podStartE2EDuration="1.575837004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:12.573093231 +0000 UTC m=+3016.612032119" watchObservedRunningTime="2026-10-30 22:57:12.575837004 +0000 UTC m=+3016.614775890"
Oct 30 22:57:13.551012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:13.550973    4581 generic.go:356] "Generic (PLEG): container finished" podID="5bd7e4d2-5b4e-415a-af27-b98a688d1724" containerID="56524657ee5a4ca2f10562b72842a9964fef3b78d12b7196c8da49d71d3b50f2" exitCode=0
Oct 30 22:57:13.551466 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:13.551047    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8" event={"ID":"5bd7e4d2-5b4e-415a-af27-b98a688d1724","Type":"ContainerDied","Data":"56524657ee5a4ca2f10562b72842a9964fef3b78d12b7196c8da49d71d3b50f2"}
Oct 30 22:57:14.286047 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.286008    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"]
Oct 30 22:57:14.286318 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.286252    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy" containerID="cri-o://4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426" gracePeriod=30
Oct 30 22:57:14.762766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.762741    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:14.908896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.908862    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd7e4d2-5b4e-415a-af27-b98a688d1724-kube-api-access-bpxk4\" (UniqueName: \"kubernetes.io/projected/5bd7e4d2-5b4e-415a-af27-b98a688d1724-kube-api-access-bpxk4\") pod \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") "
Oct 30 22:57:14.909139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.909121    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-openshift-ca\") pod \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") "
Oct 30 22:57:14.909196 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.909180    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-xks-ca\") pod \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\" (UID: \"5bd7e4d2-5b4e-415a-af27-b98a688d1724\") "
Oct 30 22:57:14.909608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.909576    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-openshift-ca" pod "5bd7e4d2-5b4e-415a-af27-b98a688d1724" (UID: "5bd7e4d2-5b4e-415a-af27-b98a688d1724"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:14.909783 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.909767    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:14.911420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.911395    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-xks-ca" pod "5bd7e4d2-5b4e-415a-af27-b98a688d1724" (UID: "5bd7e4d2-5b4e-415a-af27-b98a688d1724"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:14.911511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:14.911394    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd7e4d2-5b4e-415a-af27-b98a688d1724-kube-api-access-bpxk4" pod "5bd7e4d2-5b4e-415a-af27-b98a688d1724" (UID: "5bd7e4d2-5b4e-415a-af27-b98a688d1724"). InnerVolumeSpecName "kube-api-access-bpxk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:15.011130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:15.011093    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd7e4d2-5b4e-415a-af27-b98a688d1724-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:15.011130 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:15.011121    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxk4\" (UniqueName: \"kubernetes.io/projected/5bd7e4d2-5b4e-415a-af27-b98a688d1724-kube-api-access-bpxk4\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:15.574819 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:15.574783    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"
Oct 30 22:57:15.575011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:15.574814    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8" event={"ID":"5bd7e4d2-5b4e-415a-af27-b98a688d1724","Type":"ContainerDied","Data":"7b7c91d4757866f9bc914584b363c03b5b913327402ac9c076ad4d32777b5313"}
Oct 30 22:57:15.575011 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:15.574877    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b7c91d4757866f9bc914584b363c03b5b913327402ac9c076ad4d32777b5313"
Oct 30 22:57:19.574757 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.574685    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:57:19.618622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.618591    4581 generic.go:356] "Generic (PLEG): container finished" podID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerID="4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426" exitCode=0
Oct 30 22:57:19.618832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.618647    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"
Oct 30 22:57:19.618832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.618684    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" event={"ID":"3edd8469-95e9-41ee-a592-9c6c20aaff03","Type":"ContainerDied","Data":"4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426"}
Oct 30 22:57:19.618832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.618749    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9" event={"ID":"3edd8469-95e9-41ee-a592-9c6c20aaff03","Type":"ContainerDied","Data":"1bf93d2ea945a286f2d72c334052aae05688e3e2e97de83d934e05d38c8582d7"}
Oct 30 22:57:19.618832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.618781    4581 scope.go:122] "RemoveContainer" containerID="4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426"
Oct 30 22:57:19.631725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.631699    4581 scope.go:122] "RemoveContainer" containerID="3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d"
Oct 30 22:57:19.650483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.650362    4581 scope.go:122] "RemoveContainer" containerID="4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426"
Oct 30 22:57:19.650754 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:57:19.650731    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426\": container with ID starting with 4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426 not found: ID does not exist" containerID="4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426"
Oct 30 22:57:19.650886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.650767    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426"} err="failed to get container status \"4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426\": rpc error: code = NotFound desc = could not find container \"4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426\": container with ID starting with 4d7d548b1811116ea341850ebbc3a99c4844974c6607b6b751c7cff3c6ae3426 not found: ID does not exist"
Oct 30 22:57:19.650886 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.650794    4581 scope.go:122] "RemoveContainer" containerID="3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d"
Oct 30 22:57:19.651190 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:57:19.651170    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d\": container with ID starting with 3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d not found: ID does not exist" containerID="3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d"
Oct 30 22:57:19.651312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.651207    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d"} err="failed to get container status \"3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d\": rpc error: code = NotFound desc = could not find container \"3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d\": container with ID starting with 3bd68e463b6b5ae9f5b0bfe3c6637e632fd21cbd0c430f69c3bb541fee921f5d not found: ID does not exist"
Oct 30 22:57:19.698238 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.698187    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-token\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-token\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.698433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.698264    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-podinfo\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.698433 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.698384    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-certs\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.698720 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.698414    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-envoy\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.698809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.698672    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-certs" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:19.698884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.698867    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edd8469-95e9-41ee-a592-9c6c20aaff03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edd8469-95e9-41ee-a592-9c6c20aaff03-istiod-ca-cert\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.699414 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.699361    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edd8469-95e9-41ee-a592-9c6c20aaff03-istiod-ca-cert" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:19.699535 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.699479    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-data\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.699815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.699777    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-data" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:19.700030 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.700010    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-kube-api-access-lvl8n\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-kube-api-access-lvl8n\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.700634 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.700614    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-socket\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.701408 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.700826    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-credential-socket\") pod \"3edd8469-95e9-41ee-a592-9c6c20aaff03\" (UID: \"3edd8469-95e9-41ee-a592-9c6c20aaff03\") "
Oct 30 22:57:19.701610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.700923    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-socket" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:19.701610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.701275    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-credential-socket" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:19.702580 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.702554    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.702680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.702583    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edd8469-95e9-41ee-a592-9c6c20aaff03-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.702680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.702596    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.702680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.702611    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.702680 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.702624    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.703071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.702993    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-envoy" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:19.703182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.703089    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-kube-api-access-lvl8n" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "kube-api-access-lvl8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:19.703182 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.703109    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-podinfo" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:57:19.703439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.703420    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-token" pod "3edd8469-95e9-41ee-a592-9c6c20aaff03" (UID: "3edd8469-95e9-41ee-a592-9c6c20aaff03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:19.804071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.804039    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.804071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.804064    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.804071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.804074    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edd8469-95e9-41ee-a592-9c6c20aaff03-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.804398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.804083    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvl8n\" (UniqueName: \"kubernetes.io/projected/3edd8469-95e9-41ee-a592-9c6c20aaff03-kube-api-access-lvl8n\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:19.964275 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.964236    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"]
Oct 30 22:57:19.969622 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:19.969588    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-x4lb9"]
Oct 30 22:57:20.098896 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:20.098860    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" path="/var/lib/kubelet/pods/3edd8469-95e9-41ee-a592-9c6c20aaff03/volumes"
Oct 30 22:57:21.870718 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:21.870677    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"]
Oct 30 22:57:21.901822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:21.901788    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"]
Oct 30 22:57:26.875257 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:26.872297    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v" podUID="257cfb11-d754-4a1b-9923-73dedd664bfd" containerName="payload-processing" containerID="cri-o://2333cbcedd71f6ae32ef963e2e329adb018c16714e4c218d43a1bd796938b5cd" gracePeriod=25
Oct 30 22:57:26.903289 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:26.903241    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76" podUID="1681e335-1250-413f-bc87-61ea076bd6d9" containerName="payload-pre-processing" containerID="cri-o://6c481f3482e4995191f2bf65b344a845b8d512e485185789b52b898613b93d5c" gracePeriod=25
Oct 30 22:57:27.801370 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.801331    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"]
Oct 30 22:57:27.802499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.802471    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy"
Oct 30 22:57:27.802499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.802497    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy"
Oct 30 22:57:27.802722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.802576    4581 cpu_manager.go:414] "containerMap: removing container" podUID="5bd7e4d2-5b4e-415a-af27-b98a688d1724" containerName="revoke-keys"
Oct 30 22:57:27.802722 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.802584    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd7e4d2-5b4e-415a-af27-b98a688d1724" containerName="revoke-keys"
Oct 30 22:57:27.802824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.802755    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd7e4d2-5b4e-415a-af27-b98a688d1724" containerName="revoke-keys"
Oct 30 22:57:27.802824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.802773    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy"
Oct 30 22:57:27.802824 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.802790    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy"
Oct 30 22:57:27.807605 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.807581    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:27.829843 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.829779    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"]
Oct 30 22:57:27.836422 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.834165    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:57:27.843873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.843654    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:27.843873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.843804    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:57:27.846725 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.844017    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:27.929804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.929769    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghm2k\" (UniqueName: \"kubernetes.io/projected/f07671ac-eade-4722-8783-afa3a5ac8b21-kube-api-access-ghm2k\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:27.930296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.929972    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:27.930296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:27.930147    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:28.031667 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.031632    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:28.031883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.031763    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm2k\" (UniqueName: \"kubernetes.io/projected/f07671ac-eade-4722-8783-afa3a5ac8b21-kube-api-access-ghm2k\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:28.031883 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.031836    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:28.032388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.032363    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:28.034429 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.034402    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:28.049936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.049908    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghm2k\" (UniqueName: \"kubernetes.io/projected/f07671ac-eade-4722-8783-afa3a5ac8b21-kube-api-access-ghm2k\") pod \"maas-api-revoke-sub-2e0eb37a-qvswf\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:28.143811 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.143782    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:28.494259 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:57:28.493103    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07671ac_eade_4722_8783_afa3a5ac8b21.slice/crio-77bbf867e8d83a9889a710442903a68ff03d2a13db2192adbaba96cef1bd25ac WatchSource:0}: Error finding container 77bbf867e8d83a9889a710442903a68ff03d2a13db2192adbaba96cef1bd25ac: Status 404 returned error can't find the container with id 77bbf867e8d83a9889a710442903a68ff03d2a13db2192adbaba96cef1bd25ac
Oct 30 22:57:28.494259 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.493155    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"]
Oct 30 22:57:28.826511 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:28.826424    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf" event={"ID":"f07671ac-eade-4722-8783-afa3a5ac8b21","Type":"ContainerStarted","Data":"77bbf867e8d83a9889a710442903a68ff03d2a13db2192adbaba96cef1bd25ac"}
Oct 30 22:57:29.837379 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:29.837290    4581 generic.go:356] "Generic (PLEG): container finished" podID="f07671ac-eade-4722-8783-afa3a5ac8b21" containerID="e542de2d24e81aadfc4cb5f7cf46e1c7ed0d326776bb700a1c2b320619afd00c" exitCode=0
Oct 30 22:57:29.837958 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:29.837379    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf" event={"ID":"f07671ac-eade-4722-8783-afa3a5ac8b21","Type":"ContainerDied","Data":"e542de2d24e81aadfc4cb5f7cf46e1c7ed0d326776bb700a1c2b320619afd00c"}
Oct 30 22:57:31.096010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.095979    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:31.192244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.191248    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07671ac-eade-4722-8783-afa3a5ac8b21-kube-api-access-ghm2k\" (UniqueName: \"kubernetes.io/projected/f07671ac-eade-4722-8783-afa3a5ac8b21-kube-api-access-ghm2k\") pod \"f07671ac-eade-4722-8783-afa3a5ac8b21\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") "
Oct 30 22:57:31.192244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.191315    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-openshift-ca\") pod \"f07671ac-eade-4722-8783-afa3a5ac8b21\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") "
Oct 30 22:57:31.192244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.191504    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-xks-ca\") pod \"f07671ac-eade-4722-8783-afa3a5ac8b21\" (UID: \"f07671ac-eade-4722-8783-afa3a5ac8b21\") "
Oct 30 22:57:31.192244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.192189    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-openshift-ca" pod "f07671ac-eade-4722-8783-afa3a5ac8b21" (UID: "f07671ac-eade-4722-8783-afa3a5ac8b21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:31.215290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.214119    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-xks-ca" pod "f07671ac-eade-4722-8783-afa3a5ac8b21" (UID: "f07671ac-eade-4722-8783-afa3a5ac8b21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:31.219159 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.216719    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07671ac-eade-4722-8783-afa3a5ac8b21-kube-api-access-ghm2k" pod "f07671ac-eade-4722-8783-afa3a5ac8b21" (UID: "f07671ac-eade-4722-8783-afa3a5ac8b21"). InnerVolumeSpecName "kube-api-access-ghm2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:31.293440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.293403    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:31.293440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.293430    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghm2k\" (UniqueName: \"kubernetes.io/projected/f07671ac-eade-4722-8783-afa3a5ac8b21-kube-api-access-ghm2k\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:31.293440 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.293442    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07671ac-eade-4722-8783-afa3a5ac8b21-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:31.861309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.860011    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"
Oct 30 22:57:31.861309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.860047    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf" event={"ID":"f07671ac-eade-4722-8783-afa3a5ac8b21","Type":"ContainerDied","Data":"77bbf867e8d83a9889a710442903a68ff03d2a13db2192adbaba96cef1bd25ac"}
Oct 30 22:57:31.861309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:31.860105    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77bbf867e8d83a9889a710442903a68ff03d2a13db2192adbaba96cef1bd25ac"
Oct 30 22:57:32.479125 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:32.479083    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v" podUID="257cfb11-d754-4a1b-9923-73dedd664bfd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:57:32.479638 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:32.479161    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76" podUID="1681e335-1250-413f-bc87-61ea076bd6d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:57:33.885481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:33.885443    4581 generic.go:356] "Generic (PLEG): container finished" podID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerID="1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2" exitCode=0
Oct 30 22:57:33.885948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:33.885746    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" event={"ID":"dd29e823-643c-4ef9-b94f-dbffd2461749","Type":"ContainerDied","Data":"1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2"}
Oct 30 22:57:33.886294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:33.886275    4581 scope.go:122] "RemoveContainer" containerID="1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2"
Oct 30 22:57:33.888462 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:33.888422    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:57:33.888602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:33.888511    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:57:33.888602 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:33.888546    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:57:34.899389 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:34.899341    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" event={"ID":"dd29e823-643c-4ef9-b94f-dbffd2461749","Type":"ContainerStarted","Data":"d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e"}
Oct 30 22:57:35.085664 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:35.085624    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:57:35.085887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:35.085843    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 22:57:35.085962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:35.085890    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 22:57:36.087245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:36.085867    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 22:57:36.087245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:36.085940    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 22:57:37.086312 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:37.086268    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 22:57:37.086484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:37.086339    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 22:57:38.088005 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.087931    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 22:57:38.088590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.088016    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 22:57:38.191057 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.190891    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"]
Oct 30 22:57:38.191884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.191763    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f07671ac-eade-4722-8783-afa3a5ac8b21" containerName="revoke-keys"
Oct 30 22:57:38.191884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.191781    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07671ac-eade-4722-8783-afa3a5ac8b21" containerName="revoke-keys"
Oct 30 22:57:38.191884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.191794    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy"
Oct 30 22:57:38.191884 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.191799    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edd8469-95e9-41ee-a592-9c6c20aaff03" containerName="istio-proxy"
Oct 30 22:57:38.192255 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.191935    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07671ac-eade-4722-8783-afa3a5ac8b21" containerName="revoke-keys"
Oct 30 22:57:38.194871 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.194850    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.207919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.207898    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:57:38.216330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.216307    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:57:38.216469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.216392    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"]
Oct 30 22:57:38.216469 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.216458    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:38.217325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.217296    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:38.230113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.230087    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.230113 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.230117    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.230355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.230167    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwrt\" (UniqueName: \"kubernetes.io/projected/09485f12-6602-4bff-a5b3-1dc43a9309c0-kube-api-access-mkwrt\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.331822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.331786    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.331822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.331827    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.332110 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.331954    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwrt\" (UniqueName: \"kubernetes.io/projected/09485f12-6602-4bff-a5b3-1dc43a9309c0-kube-api-access-mkwrt\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.332539 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.332515    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.334465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.334444    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.387279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.386442    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwrt\" (UniqueName: \"kubernetes.io/projected/09485f12-6602-4bff-a5b3-1dc43a9309c0-kube-api-access-mkwrt\") pod \"maas-api-revoke-sub-79790862-7b9ms\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.515399 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.515357    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:38.892694 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.891891    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"]
Oct 30 22:57:38.939338 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:38.939311    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms" event={"ID":"09485f12-6602-4bff-a5b3-1dc43a9309c0","Type":"ContainerStarted","Data":"e61dcce27d124e7fb259cd8e3cbc91b3b54de85368f975636f973489d5df99dd"}
Oct 30 22:57:39.088267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:39.085827    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 22:57:39.088267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:39.085894    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 22:57:39.949830 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:39.949797    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms" event={"ID":"09485f12-6602-4bff-a5b3-1dc43a9309c0","Type":"ContainerStarted","Data":"44dd473da08842e1a93f7009e203219706763e4d849b34c7970abffc4af8bc0a"}
Oct 30 22:57:39.996597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:39.996549    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms" podStartSLOduration=1.996533569 podStartE2EDuration="1.996533569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:39.992520016 +0000 UTC m=+3044.031458901" watchObservedRunningTime="2026-10-30 22:57:39.996533569 +0000 UTC m=+3044.035472507"
Oct 30 22:57:40.092071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:40.088370    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 22:57:40.092071 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:40.088444    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 22:57:40.820134 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:57:40.820085    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:57:40.960890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:40.960847    4581 generic.go:356] "Generic (PLEG): container finished" podID="09485f12-6602-4bff-a5b3-1dc43a9309c0" containerID="44dd473da08842e1a93f7009e203219706763e4d849b34c7970abffc4af8bc0a" exitCode=0
Oct 30 22:57:40.961092 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:40.960922    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms" event={"ID":"09485f12-6602-4bff-a5b3-1dc43a9309c0","Type":"ContainerDied","Data":"44dd473da08842e1a93f7009e203219706763e4d849b34c7970abffc4af8bc0a"}
Oct 30 22:57:41.092038 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:41.091950    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:57:41.092372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:41.092350    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:57:41.093253 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:41.093195    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 22:57:42.170711 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.170680    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:42.227936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.227839    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-xks-ca\") pod \"09485f12-6602-4bff-a5b3-1dc43a9309c0\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") "
Oct 30 22:57:42.227936 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.227892    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09485f12-6602-4bff-a5b3-1dc43a9309c0-kube-api-access-mkwrt\" (UniqueName: \"kubernetes.io/projected/09485f12-6602-4bff-a5b3-1dc43a9309c0-kube-api-access-mkwrt\") pod \"09485f12-6602-4bff-a5b3-1dc43a9309c0\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") "
Oct 30 22:57:42.228192 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.227955    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-openshift-ca\") pod \"09485f12-6602-4bff-a5b3-1dc43a9309c0\" (UID: \"09485f12-6602-4bff-a5b3-1dc43a9309c0\") "
Oct 30 22:57:42.228777 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.228742    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-openshift-ca" pod "09485f12-6602-4bff-a5b3-1dc43a9309c0" (UID: "09485f12-6602-4bff-a5b3-1dc43a9309c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:42.230577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.230546    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09485f12-6602-4bff-a5b3-1dc43a9309c0-kube-api-access-mkwrt" pod "09485f12-6602-4bff-a5b3-1dc43a9309c0" (UID: "09485f12-6602-4bff-a5b3-1dc43a9309c0"). InnerVolumeSpecName "kube-api-access-mkwrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:42.230697 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.230634    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-xks-ca" pod "09485f12-6602-4bff-a5b3-1dc43a9309c0" (UID: "09485f12-6602-4bff-a5b3-1dc43a9309c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:42.330481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.330452    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:42.330481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.330480    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwrt\" (UniqueName: \"kubernetes.io/projected/09485f12-6602-4bff-a5b3-1dc43a9309c0-kube-api-access-mkwrt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:42.330702 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.330493    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09485f12-6602-4bff-a5b3-1dc43a9309c0-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:42.478919 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.478830    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v" podUID="257cfb11-d754-4a1b-9923-73dedd664bfd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:57:42.479085 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.478836    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76" podUID="1681e335-1250-413f-bc87-61ea076bd6d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:57:42.992542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.992512    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"
Oct 30 22:57:42.992542 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.992530    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms" event={"ID":"09485f12-6602-4bff-a5b3-1dc43a9309c0","Type":"ContainerDied","Data":"e61dcce27d124e7fb259cd8e3cbc91b3b54de85368f975636f973489d5df99dd"}
Oct 30 22:57:42.992809 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:42.992588    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e61dcce27d124e7fb259cd8e3cbc91b3b54de85368f975636f973489d5df99dd"
Oct 30 22:57:47.037165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.036501    4581 generic.go:356] "Generic (PLEG): container finished" podID="257cfb11-d754-4a1b-9923-73dedd664bfd" containerID="2333cbcedd71f6ae32ef963e2e329adb018c16714e4c218d43a1bd796938b5cd" exitCode=0
Oct 30 22:57:47.037165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.036701    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v" event={"ID":"257cfb11-d754-4a1b-9923-73dedd664bfd","Type":"ContainerDied","Data":"2333cbcedd71f6ae32ef963e2e329adb018c16714e4c218d43a1bd796938b5cd"}
Oct 30 22:57:47.039782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.039605    4581 generic.go:356] "Generic (PLEG): container finished" podID="1681e335-1250-413f-bc87-61ea076bd6d9" containerID="6c481f3482e4995191f2bf65b344a845b8d512e485185789b52b898613b93d5c" exitCode=0
Oct 30 22:57:47.039782 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.039733    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76" event={"ID":"1681e335-1250-413f-bc87-61ea076bd6d9","Type":"ContainerDied","Data":"6c481f3482e4995191f2bf65b344a845b8d512e485185789b52b898613b93d5c"}
Oct 30 22:57:47.214554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.214525    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:57:47.273393 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.273352    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:57:47.314154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.313152    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257cfb11-d754-4a1b-9923-73dedd664bfd-kube-api-access-tnwns\" (UniqueName: \"kubernetes.io/projected/257cfb11-d754-4a1b-9923-73dedd664bfd-kube-api-access-tnwns\") pod \"257cfb11-d754-4a1b-9923-73dedd664bfd\" (UID: \"257cfb11-d754-4a1b-9923-73dedd664bfd\") "
Oct 30 22:57:47.314154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.313316    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257cfb11-d754-4a1b-9923-73dedd664bfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257cfb11-d754-4a1b-9923-73dedd664bfd-plugins-config-volume\") pod \"257cfb11-d754-4a1b-9923-73dedd664bfd\" (UID: \"257cfb11-d754-4a1b-9923-73dedd664bfd\") "
Oct 30 22:57:47.321270 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.314586    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257cfb11-d754-4a1b-9923-73dedd664bfd-plugins-config-volume" pod "257cfb11-d754-4a1b-9923-73dedd664bfd" (UID: "257cfb11-d754-4a1b-9923-73dedd664bfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:47.326291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.326205    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257cfb11-d754-4a1b-9923-73dedd664bfd-kube-api-access-tnwns" pod "257cfb11-d754-4a1b-9923-73dedd664bfd" (UID: "257cfb11-d754-4a1b-9923-73dedd664bfd"). InnerVolumeSpecName "kube-api-access-tnwns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:47.414629 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.414555    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1681e335-1250-413f-bc87-61ea076bd6d9-kube-api-access-nr8kg\" (UniqueName: \"kubernetes.io/projected/1681e335-1250-413f-bc87-61ea076bd6d9-kube-api-access-nr8kg\") pod \"1681e335-1250-413f-bc87-61ea076bd6d9\" (UID: \"1681e335-1250-413f-bc87-61ea076bd6d9\") "
Oct 30 22:57:47.414822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.414658    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1681e335-1250-413f-bc87-61ea076bd6d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1681e335-1250-413f-bc87-61ea076bd6d9-plugins-config-volume\") pod \"1681e335-1250-413f-bc87-61ea076bd6d9\" (UID: \"1681e335-1250-413f-bc87-61ea076bd6d9\") "
Oct 30 22:57:47.415023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.414997    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1681e335-1250-413f-bc87-61ea076bd6d9-plugins-config-volume" pod "1681e335-1250-413f-bc87-61ea076bd6d9" (UID: "1681e335-1250-413f-bc87-61ea076bd6d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:47.415262 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.415196    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnwns\" (UniqueName: \"kubernetes.io/projected/257cfb11-d754-4a1b-9923-73dedd664bfd-kube-api-access-tnwns\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:47.415342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.415272    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/257cfb11-d754-4a1b-9923-73dedd664bfd-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:47.415342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.415289    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1681e335-1250-413f-bc87-61ea076bd6d9-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:47.416848 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.416826    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1681e335-1250-413f-bc87-61ea076bd6d9-kube-api-access-nr8kg" pod "1681e335-1250-413f-bc87-61ea076bd6d9" (UID: "1681e335-1250-413f-bc87-61ea076bd6d9"). InnerVolumeSpecName "kube-api-access-nr8kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:47.517412 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:47.517376    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8kg\" (UniqueName: \"kubernetes.io/projected/1681e335-1250-413f-bc87-61ea076bd6d9-kube-api-access-nr8kg\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:48.051187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.051158    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"
Oct 30 22:57:48.051636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.051185    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v" event={"ID":"257cfb11-d754-4a1b-9923-73dedd664bfd","Type":"ContainerDied","Data":"748dd2ed457a8e778e2c2cc33232e9066e0621ad404332dabe9cc77b105efa50"}
Oct 30 22:57:48.051636 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.051304    4581 scope.go:122] "RemoveContainer" containerID="2333cbcedd71f6ae32ef963e2e329adb018c16714e4c218d43a1bd796938b5cd"
Oct 30 22:57:48.053040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.052975    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"
Oct 30 22:57:48.053040 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.052993    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76" event={"ID":"1681e335-1250-413f-bc87-61ea076bd6d9","Type":"ContainerDied","Data":"d8574eaf652aa1ffd25702de3a7b99d96bf04c283f04e7b6d82354e30a2100c5"}
Oct 30 22:57:48.065903 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.065883    4581 scope.go:122] "RemoveContainer" containerID="6c481f3482e4995191f2bf65b344a845b8d512e485185789b52b898613b93d5c"
Oct 30 22:57:48.111484 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.111459    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"]
Oct 30 22:57:48.116486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.116463    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b03b33-7c5fcd4d57-g2t4v"]
Oct 30 22:57:48.148512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.148483    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"]
Oct 30 22:57:48.153394 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:48.153331    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b03b33-cd67bd8f5-dft76"]
Oct 30 22:57:50.107620 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:50.107584    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1681e335-1250-413f-bc87-61ea076bd6d9" path="/var/lib/kubelet/pods/1681e335-1250-413f-bc87-61ea076bd6d9/volumes"
Oct 30 22:57:50.108058 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:50.108042    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257cfb11-d754-4a1b-9923-73dedd664bfd" path="/var/lib/kubelet/pods/257cfb11-d754-4a1b-9923-73dedd664bfd/volumes"
Oct 30 22:57:50.825519 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:50.825488    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"]
Oct 30 22:57:55.826269 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:55.826201    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws" podUID="2daea72b-b631-4216-a051-8155f0c0f1c1" containerName="payload-pre-processing" containerID="cri-o://0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6" gracePeriod=25
Oct 30 22:57:56.131456 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.131424    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:57:56.137922 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.137891    4581 generic.go:356] "Generic (PLEG): container finished" podID="2daea72b-b631-4216-a051-8155f0c0f1c1" containerID="0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6" exitCode=0
Oct 30 22:57:56.138115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.138008    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws" event={"ID":"2daea72b-b631-4216-a051-8155f0c0f1c1","Type":"ContainerDied","Data":"0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6"}
Oct 30 22:57:56.138115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.138050    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws" event={"ID":"2daea72b-b631-4216-a051-8155f0c0f1c1","Type":"ContainerDied","Data":"3b30901799495161fcf3814349ba083a21b3ff24e7bd1cf3937085f4e626ce0c"}
Oct 30 22:57:56.138115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.138075    4581 scope.go:122] "RemoveContainer" containerID="0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6"
Oct 30 22:57:56.138115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.138084    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"
Oct 30 22:57:56.154342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.154192    4581 scope.go:122] "RemoveContainer" containerID="0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6"
Oct 30 22:57:56.154639 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:57:56.154610    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6\": container with ID starting with 0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6 not found: ID does not exist" containerID="0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6"
Oct 30 22:57:56.154717 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.154652    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6"} err="failed to get container status \"0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6\": rpc error: code = NotFound desc = could not find container \"0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6\": container with ID starting with 0637308333c8d81666d1ab3355d6eec994fad63ff8debe1909e0a1f6f81015d6 not found: ID does not exist"
Oct 30 22:57:56.163968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.162316    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2daea72b-b631-4216-a051-8155f0c0f1c1-kube-api-access-bnkk9\" (UniqueName: \"kubernetes.io/projected/2daea72b-b631-4216-a051-8155f0c0f1c1-kube-api-access-bnkk9\") pod \"2daea72b-b631-4216-a051-8155f0c0f1c1\" (UID: \"2daea72b-b631-4216-a051-8155f0c0f1c1\") "
Oct 30 22:57:56.163968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.162700    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2daea72b-b631-4216-a051-8155f0c0f1c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daea72b-b631-4216-a051-8155f0c0f1c1-plugins-config-volume\") pod \"2daea72b-b631-4216-a051-8155f0c0f1c1\" (UID: \"2daea72b-b631-4216-a051-8155f0c0f1c1\") "
Oct 30 22:57:56.163968 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.163757    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2daea72b-b631-4216-a051-8155f0c0f1c1-plugins-config-volume" pod "2daea72b-b631-4216-a051-8155f0c0f1c1" (UID: "2daea72b-b631-4216-a051-8155f0c0f1c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:56.167014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.166983    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daea72b-b631-4216-a051-8155f0c0f1c1-kube-api-access-bnkk9" pod "2daea72b-b631-4216-a051-8155f0c0f1c1" (UID: "2daea72b-b631-4216-a051-8155f0c0f1c1"). InnerVolumeSpecName "kube-api-access-bnkk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:56.264964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.264929    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daea72b-b631-4216-a051-8155f0c0f1c1-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:56.264964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.264961    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnkk9\" (UniqueName: \"kubernetes.io/projected/2daea72b-b631-4216-a051-8155f0c0f1c1-kube-api-access-bnkk9\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:57:56.481014 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.480983    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"]
Oct 30 22:57:56.490874 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:56.490845    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0a807-78f9946d59-tjxws"]
Oct 30 22:57:58.100508 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:57:58.100474    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2daea72b-b631-4216-a051-8155f0c0f1c1" path="/var/lib/kubelet/pods/2daea72b-b631-4216-a051-8155f0c0f1c1/volumes"
Oct 30 22:58:06.615499 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:06.615463    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"]
Oct 30 22:58:06.624483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:06.624443    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b03b33-gvnl8"]
Oct 30 22:58:08.098313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:08.098270    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd7e4d2-5b4e-415a-af27-b98a688d1724" path="/var/lib/kubelet/pods/5bd7e4d2-5b4e-415a-af27-b98a688d1724/volumes"
Oct 30 22:58:10.820113 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:58:10.820078    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:58:14.101630 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.101599    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"]
Oct 30 22:58:14.102424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102399    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1681e335-1250-413f-bc87-61ea076bd6d9" containerName="payload-pre-processing"
Oct 30 22:58:14.102424 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102425    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1681e335-1250-413f-bc87-61ea076bd6d9" containerName="payload-pre-processing"
Oct 30 22:58:14.102643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102452    4581 cpu_manager.go:414] "containerMap: removing container" podUID="09485f12-6602-4bff-a5b3-1dc43a9309c0" containerName="revoke-keys"
Oct 30 22:58:14.102643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102460    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09485f12-6602-4bff-a5b3-1dc43a9309c0" containerName="revoke-keys"
Oct 30 22:58:14.102643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102477    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2daea72b-b631-4216-a051-8155f0c0f1c1" containerName="payload-pre-processing"
Oct 30 22:58:14.102643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102482    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2daea72b-b631-4216-a051-8155f0c0f1c1" containerName="payload-pre-processing"
Oct 30 22:58:14.102643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102517    4581 cpu_manager.go:414] "containerMap: removing container" podUID="257cfb11-d754-4a1b-9923-73dedd664bfd" containerName="payload-processing"
Oct 30 22:58:14.102643 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102522    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257cfb11-d754-4a1b-9923-73dedd664bfd" containerName="payload-processing"
Oct 30 22:58:14.102982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102686    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1681e335-1250-413f-bc87-61ea076bd6d9" containerName="payload-pre-processing"
Oct 30 22:58:14.102982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102707    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="09485f12-6602-4bff-a5b3-1dc43a9309c0" containerName="revoke-keys"
Oct 30 22:58:14.102982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102717    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2daea72b-b631-4216-a051-8155f0c0f1c1" containerName="payload-pre-processing"
Oct 30 22:58:14.102982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.102732    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="257cfb11-d754-4a1b-9923-73dedd664bfd" containerName="payload-processing"
Oct 30 22:58:14.108398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.108376    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.118315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.117872    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"]
Oct 30 22:58:14.129881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.129856    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:58:14.130163 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.130147    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:58:14.207526 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.207494    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:58:14.207780 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.207514    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:58:14.219157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.219124    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.219374 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.219186    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frknl\" (UniqueName: \"kubernetes.io/projected/7c37a12c-492f-404a-9582-c409842fb43d-kube-api-access-frknl\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.219455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.219369    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.320954 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.320918    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.321154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.320971    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frknl\" (UniqueName: \"kubernetes.io/projected/7c37a12c-492f-404a-9582-c409842fb43d-kube-api-access-frknl\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.321154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.321024    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.321815 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.321733    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.323565 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.323539    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.339594 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.339568    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frknl\" (UniqueName: \"kubernetes.io/projected/7c37a12c-492f-404a-9582-c409842fb43d-kube-api-access-frknl\") pod \"maas-api-revoke-sub-8cb67f98-w58lr\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.395763 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.395731    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"]
Oct 30 22:58:14.400672 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.400648    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.421315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.421276    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"]
Oct 30 22:58:14.427677 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.427642    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:14.524373 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.523660    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.525512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.525074    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.525512 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.525184    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqls\" (UniqueName: \"kubernetes.io/projected/b225e238-bde0-41f4-916b-2dbb557c0630-kube-api-access-fvqls\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.626516 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.626484    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.626734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.626526    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.626734 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.626577    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqls\" (UniqueName: \"kubernetes.io/projected/b225e238-bde0-41f4-916b-2dbb557c0630-kube-api-access-fvqls\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.627282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.627259    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.629187 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.629158    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.644559 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.644535    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqls\" (UniqueName: \"kubernetes.io/projected/b225e238-bde0-41f4-916b-2dbb557c0630-kube-api-access-fvqls\") pod \"maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.716595 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.716518    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:14.761892 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:14.761860    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"]
Oct 30 22:58:14.772901 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:58:14.772863    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c37a12c_492f_404a_9582_c409842fb43d.slice/crio-1616815904f094a98cde0dbf20dc8bc792368f4c681db83187545872557b99a7 WatchSource:0}: Error finding container 1616815904f094a98cde0dbf20dc8bc792368f4c681db83187545872557b99a7: Status 404 returned error can't find the container with id 1616815904f094a98cde0dbf20dc8bc792368f4c681db83187545872557b99a7
Oct 30 22:58:15.245380 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:15.245356    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"]
Oct 30 22:58:15.247178 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:58:15.247152    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb225e238_bde0_41f4_916b_2dbb557c0630.slice/crio-c6103c3970a74435929c84dc7b058e023089e7f3e644777c3f0f05be98ecc843 WatchSource:0}: Error finding container c6103c3970a74435929c84dc7b058e023089e7f3e644777c3f0f05be98ecc843: Status 404 returned error can't find the container with id c6103c3970a74435929c84dc7b058e023089e7f3e644777c3f0f05be98ecc843
Oct 30 22:58:15.327007 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:15.326961    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp" event={"ID":"b225e238-bde0-41f4-916b-2dbb557c0630","Type":"ContainerStarted","Data":"c6103c3970a74435929c84dc7b058e023089e7f3e644777c3f0f05be98ecc843"}
Oct 30 22:58:15.328859 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:15.328832    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr" event={"ID":"7c37a12c-492f-404a-9582-c409842fb43d","Type":"ContainerStarted","Data":"1616815904f094a98cde0dbf20dc8bc792368f4c681db83187545872557b99a7"}
Oct 30 22:58:16.342885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:16.342839    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp" event={"ID":"b225e238-bde0-41f4-916b-2dbb557c0630","Type":"ContainerStarted","Data":"f0ed13c6f491faf149457f1b5b114082671c4ed695db6b9d45410db440545961"}
Oct 30 22:58:16.345715 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:16.345680    4581 generic.go:356] "Generic (PLEG): container finished" podID="7c37a12c-492f-404a-9582-c409842fb43d" containerID="ce8bbdc8167d0fc51e6cb82b15087d271f19dc5e71aad7905a6e0d476c975aa3" exitCode=0
Oct 30 22:58:16.345982 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:16.345961    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr" event={"ID":"7c37a12c-492f-404a-9582-c409842fb43d","Type":"ContainerDied","Data":"ce8bbdc8167d0fc51e6cb82b15087d271f19dc5e71aad7905a6e0d476c975aa3"}
Oct 30 22:58:16.398243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:16.387458    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp" podStartSLOduration=2.387438741 podStartE2EDuration="2.387438741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:16.38678289 +0000 UTC m=+3080.425721783" watchObservedRunningTime="2026-10-30 22:58:16.387438741 +0000 UTC m=+3080.426377619"
Oct 30 22:58:17.366949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.364989    4581 generic.go:356] "Generic (PLEG): container finished" podID="b225e238-bde0-41f4-916b-2dbb557c0630" containerID="f0ed13c6f491faf149457f1b5b114082671c4ed695db6b9d45410db440545961" exitCode=0
Oct 30 22:58:17.366949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.365667    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp" event={"ID":"b225e238-bde0-41f4-916b-2dbb557c0630","Type":"ContainerDied","Data":"f0ed13c6f491faf149457f1b5b114082671c4ed695db6b9d45410db440545961"}
Oct 30 22:58:17.703703 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.703673    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:17.781501 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.781466    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c37a12c-492f-404a-9582-c409842fb43d-kube-api-access-frknl\" (UniqueName: \"kubernetes.io/projected/7c37a12c-492f-404a-9582-c409842fb43d-kube-api-access-frknl\") pod \"7c37a12c-492f-404a-9582-c409842fb43d\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") "
Oct 30 22:58:17.781810 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.781787    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-xks-ca\") pod \"7c37a12c-492f-404a-9582-c409842fb43d\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") "
Oct 30 22:58:17.782066 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.782046    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-openshift-ca\") pod \"7c37a12c-492f-404a-9582-c409842fb43d\" (UID: \"7c37a12c-492f-404a-9582-c409842fb43d\") "
Oct 30 22:58:17.782690 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.782660    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-openshift-ca" pod "7c37a12c-492f-404a-9582-c409842fb43d" (UID: "7c37a12c-492f-404a-9582-c409842fb43d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:17.787420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.784591    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c37a12c-492f-404a-9582-c409842fb43d-kube-api-access-frknl" pod "7c37a12c-492f-404a-9582-c409842fb43d" (UID: "7c37a12c-492f-404a-9582-c409842fb43d"). InnerVolumeSpecName "kube-api-access-frknl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:17.787420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.787361    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:17.789580 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.789555    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-xks-ca" pod "7c37a12c-492f-404a-9582-c409842fb43d" (UID: "7c37a12c-492f-404a-9582-c409842fb43d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:17.889669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.889564    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frknl\" (UniqueName: \"kubernetes.io/projected/7c37a12c-492f-404a-9582-c409842fb43d-kube-api-access-frknl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:17.889669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:17.889604    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c37a12c-492f-404a-9582-c409842fb43d-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:18.378623 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.378588    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"
Oct 30 22:58:18.379158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.378615    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr" event={"ID":"7c37a12c-492f-404a-9582-c409842fb43d","Type":"ContainerDied","Data":"1616815904f094a98cde0dbf20dc8bc792368f4c681db83187545872557b99a7"}
Oct 30 22:58:18.379158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.378671    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1616815904f094a98cde0dbf20dc8bc792368f4c681db83187545872557b99a7"
Oct 30 22:58:18.612561 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.612531    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:18.713237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.712388    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b225e238-bde0-41f4-916b-2dbb557c0630-kube-api-access-fvqls\" (UniqueName: \"kubernetes.io/projected/b225e238-bde0-41f4-916b-2dbb557c0630-kube-api-access-fvqls\") pod \"b225e238-bde0-41f4-916b-2dbb557c0630\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") "
Oct 30 22:58:18.713237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.712528    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-openshift-ca\") pod \"b225e238-bde0-41f4-916b-2dbb557c0630\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") "
Oct 30 22:58:18.713237 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.712699    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-xks-ca\") pod \"b225e238-bde0-41f4-916b-2dbb557c0630\" (UID: \"b225e238-bde0-41f4-916b-2dbb557c0630\") "
Oct 30 22:58:18.726239 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.722380    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-openshift-ca" pod "b225e238-bde0-41f4-916b-2dbb557c0630" (UID: "b225e238-bde0-41f4-916b-2dbb557c0630"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:18.735009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.730042    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:18.737315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.736362    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-xks-ca" pod "b225e238-bde0-41f4-916b-2dbb557c0630" (UID: "b225e238-bde0-41f4-916b-2dbb557c0630"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:18.737315 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.737275    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b225e238-bde0-41f4-916b-2dbb557c0630-kube-api-access-fvqls" pod "b225e238-bde0-41f4-916b-2dbb557c0630" (UID: "b225e238-bde0-41f4-916b-2dbb557c0630"). InnerVolumeSpecName "kube-api-access-fvqls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:18.831947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.831910    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqls\" (UniqueName: \"kubernetes.io/projected/b225e238-bde0-41f4-916b-2dbb557c0630-kube-api-access-fvqls\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:18.831947 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:18.831948    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225e238-bde0-41f4-916b-2dbb557c0630-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:19.391838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:19.391797    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp" event={"ID":"b225e238-bde0-41f4-916b-2dbb557c0630","Type":"ContainerDied","Data":"c6103c3970a74435929c84dc7b058e023089e7f3e644777c3f0f05be98ecc843"}
Oct 30 22:58:19.391838 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:19.391836    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"
Oct 30 22:58:19.392448 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:19.391935    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6103c3970a74435929c84dc7b058e023089e7f3e644777c3f0f05be98ecc843"
Oct 30 22:58:24.422231 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.422187    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"]
Oct 30 22:58:24.422908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.422890    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7c37a12c-492f-404a-9582-c409842fb43d" containerName="revoke-keys"
Oct 30 22:58:24.422908 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.422908    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c37a12c-492f-404a-9582-c409842fb43d" containerName="revoke-keys"
Oct 30 22:58:24.423043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.422950    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b225e238-bde0-41f4-916b-2dbb557c0630" containerName="revoke-keys"
Oct 30 22:58:24.423043 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.422957    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b225e238-bde0-41f4-916b-2dbb557c0630" containerName="revoke-keys"
Oct 30 22:58:24.423139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.423056    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b225e238-bde0-41f4-916b-2dbb557c0630" containerName="revoke-keys"
Oct 30 22:58:24.423139 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.423068    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c37a12c-492f-404a-9582-c409842fb43d" containerName="revoke-keys"
Oct 30 22:58:24.427419 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.427398    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.450095 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.450069    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:58:24.450281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.450086    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:58:24.450281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.450110    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:58:24.450281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.450131    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:58:24.458356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.458328    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"]
Oct 30 22:58:24.524748 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.524702    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqs6\" (UniqueName: \"kubernetes.io/projected/08507474-f98e-45eb-a390-a6aa9f21b39e-kube-api-access-8gqs6\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.524938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.524765    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.525103 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.525075    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.626695 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.626659    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqs6\" (UniqueName: \"kubernetes.io/projected/08507474-f98e-45eb-a390-a6aa9f21b39e-kube-api-access-8gqs6\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.626897 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.626712    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.626955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.626900    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.627438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.627410    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.629590 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.629568    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.644925 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.644887    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqs6\" (UniqueName: \"kubernetes.io/projected/08507474-f98e-45eb-a390-a6aa9f21b39e-kube-api-access-8gqs6\") pod \"maas-api-revoke-sub-0605cb02-zbgrj\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.745432 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.745310    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:24.861619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.861573    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"]
Oct 30 22:58:24.911116 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:24.911079    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"]
Oct 30 22:58:25.065390 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:25.065358    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"]
Oct 30 22:58:25.066364 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:58:25.066338    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08507474_f98e_45eb_a390_a6aa9f21b39e.slice/crio-473fcd6d8d4a27a52274f2bec9617ed6bec014644887c9d12765415c52abe58f WatchSource:0}: Error finding container 473fcd6d8d4a27a52274f2bec9617ed6bec014644887c9d12765415c52abe58f: Status 404 returned error can't find the container with id 473fcd6d8d4a27a52274f2bec9617ed6bec014644887c9d12765415c52abe58f
Oct 30 22:58:25.442086 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:25.442058    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls_9142b111-3a39-4408-9784-52eda5706504/payload-processing/0.log"
Oct 30 22:58:25.447986 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:25.447954    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj" event={"ID":"08507474-f98e-45eb-a390-a6aa9f21b39e","Type":"ContainerStarted","Data":"473fcd6d8d4a27a52274f2bec9617ed6bec014644887c9d12765415c52abe58f"}
Oct 30 22:58:25.656881 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:25.656853    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls_9142b111-3a39-4408-9784-52eda5706504/payload-processing/0.log"
Oct 30 22:58:26.395120 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.395079    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"]
Oct 30 22:58:26.399771 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.399747    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.419135 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.419105    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"]
Oct 30 22:58:26.456364 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.456329    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjf7x\" (UniqueName: \"kubernetes.io/projected/d372c52d-b463-4e63-9df1-9c21d967e53c-kube-api-access-tjf7x\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.456766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.456509    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.456766 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.456581    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.460365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.460337    4581 generic.go:356] "Generic (PLEG): container finished" podID="08507474-f98e-45eb-a390-a6aa9f21b39e" containerID="225ca671be30c0a4d4b5efff6868a29077639724b4b5a62cbdbce77f19b8ceab" exitCode=0
Oct 30 22:58:26.460530 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.460369    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj" event={"ID":"08507474-f98e-45eb-a390-a6aa9f21b39e","Type":"ContainerDied","Data":"225ca671be30c0a4d4b5efff6868a29077639724b4b5a62cbdbce77f19b8ceab"}
Oct 30 22:58:26.558601 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.558553    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjf7x\" (UniqueName: \"kubernetes.io/projected/d372c52d-b463-4e63-9df1-9c21d967e53c-kube-api-access-tjf7x\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.558788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.558665    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.558788 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.558707    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.559823 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.559791    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.572851 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.572814    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.576037 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.576006    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjf7x\" (UniqueName: \"kubernetes.io/projected/d372c52d-b463-4e63-9df1-9c21d967e53c-kube-api-access-tjf7x\") pod \"maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:26.742640 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:26.742609    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:27.062040 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:58:27.062010    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd372c52d_b463_4e63_9df1_9c21d967e53c.slice/crio-9e214abf054d0c9724d11fb756101f33bb89af4032665ee411c3e55c659ffa67 WatchSource:0}: Error finding container 9e214abf054d0c9724d11fb756101f33bb89af4032665ee411c3e55c659ffa67: Status 404 returned error can't find the container with id 9e214abf054d0c9724d11fb756101f33bb89af4032665ee411c3e55c659ffa67
Oct 30 22:58:27.062242 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.062199    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"]
Oct 30 22:58:27.471714 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.471683    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn" event={"ID":"d372c52d-b463-4e63-9df1-9c21d967e53c","Type":"ContainerStarted","Data":"9e214abf054d0c9724d11fb756101f33bb89af4032665ee411c3e55c659ffa67"}
Oct 30 22:58:27.642129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.642097    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:27.775825 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.775797    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-xks-ca\") pod \"08507474-f98e-45eb-a390-a6aa9f21b39e\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") "
Oct 30 22:58:27.776004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.775857    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08507474-f98e-45eb-a390-a6aa9f21b39e-kube-api-access-8gqs6\" (UniqueName: \"kubernetes.io/projected/08507474-f98e-45eb-a390-a6aa9f21b39e-kube-api-access-8gqs6\") pod \"08507474-f98e-45eb-a390-a6aa9f21b39e\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") "
Oct 30 22:58:27.776004 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.775907    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-openshift-ca\") pod \"08507474-f98e-45eb-a390-a6aa9f21b39e\" (UID: \"08507474-f98e-45eb-a390-a6aa9f21b39e\") "
Oct 30 22:58:27.776375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.776345    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-openshift-ca" pod "08507474-f98e-45eb-a390-a6aa9f21b39e" (UID: "08507474-f98e-45eb-a390-a6aa9f21b39e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:27.778287 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.778251    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-xks-ca" pod "08507474-f98e-45eb-a390-a6aa9f21b39e" (UID: "08507474-f98e-45eb-a390-a6aa9f21b39e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:27.778545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.778520    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08507474-f98e-45eb-a390-a6aa9f21b39e-kube-api-access-8gqs6" pod "08507474-f98e-45eb-a390-a6aa9f21b39e" (UID: "08507474-f98e-45eb-a390-a6aa9f21b39e"). InnerVolumeSpecName "kube-api-access-8gqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:27.879016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.878918    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:27.879016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.878955    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqs6\" (UniqueName: \"kubernetes.io/projected/08507474-f98e-45eb-a390-a6aa9f21b39e-kube-api-access-8gqs6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:27.879016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:27.878968    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08507474-f98e-45eb-a390-a6aa9f21b39e-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:28.484246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:28.484122    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"
Oct 30 22:58:28.484246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:28.484145    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj" event={"ID":"08507474-f98e-45eb-a390-a6aa9f21b39e","Type":"ContainerDied","Data":"473fcd6d8d4a27a52274f2bec9617ed6bec014644887c9d12765415c52abe58f"}
Oct 30 22:58:28.484246 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:28.484212    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="473fcd6d8d4a27a52274f2bec9617ed6bec014644887c9d12765415c52abe58f"
Oct 30 22:58:28.486284 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:28.486254    4581 generic.go:356] "Generic (PLEG): container finished" podID="d372c52d-b463-4e63-9df1-9c21d967e53c" containerID="a8ee0e9f622022d3a5ba528ef1b6486d2c0985c924d71319a1f6f583c181f694" exitCode=0
Oct 30 22:58:28.486707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:28.486678    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn" event={"ID":"d372c52d-b463-4e63-9df1-9c21d967e53c","Type":"ContainerDied","Data":"a8ee0e9f622022d3a5ba528ef1b6486d2c0985c924d71319a1f6f583c181f694"}
Oct 30 22:58:29.674537 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.674511    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:29.806568 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.806480    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d372c52d-b463-4e63-9df1-9c21d967e53c-kube-api-access-tjf7x\" (UniqueName: \"kubernetes.io/projected/d372c52d-b463-4e63-9df1-9c21d967e53c-kube-api-access-tjf7x\") pod \"d372c52d-b463-4e63-9df1-9c21d967e53c\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") "
Oct 30 22:58:29.806759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.806640    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-openshift-ca\") pod \"d372c52d-b463-4e63-9df1-9c21d967e53c\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") "
Oct 30 22:58:29.806759 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.806755    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-xks-ca\") pod \"d372c52d-b463-4e63-9df1-9c21d967e53c\" (UID: \"d372c52d-b463-4e63-9df1-9c21d967e53c\") "
Oct 30 22:58:29.807032 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.807002    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-openshift-ca" pod "d372c52d-b463-4e63-9df1-9c21d967e53c" (UID: "d372c52d-b463-4e63-9df1-9c21d967e53c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:29.807361 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.807339    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:29.808879 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.808855    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d372c52d-b463-4e63-9df1-9c21d967e53c-kube-api-access-tjf7x" pod "d372c52d-b463-4e63-9df1-9c21d967e53c" (UID: "d372c52d-b463-4e63-9df1-9c21d967e53c"). InnerVolumeSpecName "kube-api-access-tjf7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:29.809000 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.808970    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-xks-ca" pod "d372c52d-b463-4e63-9df1-9c21d967e53c" (UID: "d372c52d-b463-4e63-9df1-9c21d967e53c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:29.862878 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.862831    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz" podUID="879ed01f-aaea-42f9-86dd-eb467e927802" containerName="payload-processing" containerID="cri-o://80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff" gracePeriod=25
Oct 30 22:58:29.909124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.909089    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372c52d-b463-4e63-9df1-9c21d967e53c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:29.909124 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.909119    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjf7x\" (UniqueName: \"kubernetes.io/projected/d372c52d-b463-4e63-9df1-9c21d967e53c-kube-api-access-tjf7x\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:29.912356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:29.912325    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp" podUID="9f048de9-6d24-4561-bfcd-d0501b41bbf2" containerName="payload-pre-processing" containerID="cri-o://b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407" gracePeriod=25
Oct 30 22:58:30.507189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:30.507098    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"
Oct 30 22:58:30.507189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:30.507130    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn" event={"ID":"d372c52d-b463-4e63-9df1-9c21d967e53c","Type":"ContainerDied","Data":"9e214abf054d0c9724d11fb756101f33bb89af4032665ee411c3e55c659ffa67"}
Oct 30 22:58:30.507189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:30.507181    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e214abf054d0c9724d11fb756101f33bb89af4032665ee411c3e55c659ffa67"
Oct 30 22:58:38.567483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:38.567398    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz" podUID="879ed01f-aaea-42f9-86dd-eb467e927802" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:58:38.567483 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:38.567458    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp" podUID="9f048de9-6d24-4561-bfcd-d0501b41bbf2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:58:40.819932 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:58:40.819891    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:58:47.301964 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.301922    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"]
Oct 30 22:58:47.302862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.302839    4581 cpu_manager.go:414] "containerMap: removing container" podUID="08507474-f98e-45eb-a390-a6aa9f21b39e" containerName="revoke-keys"
Oct 30 22:58:47.302977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.302864    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08507474-f98e-45eb-a390-a6aa9f21b39e" containerName="revoke-keys"
Oct 30 22:58:47.302977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.302953    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d372c52d-b463-4e63-9df1-9c21d967e53c" containerName="revoke-keys"
Oct 30 22:58:47.302977 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.302962    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d372c52d-b463-4e63-9df1-9c21d967e53c" containerName="revoke-keys"
Oct 30 22:58:47.303154 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.303134    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="08507474-f98e-45eb-a390-a6aa9f21b39e" containerName="revoke-keys"
Oct 30 22:58:47.303254 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.303163    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d372c52d-b463-4e63-9df1-9c21d967e53c" containerName="revoke-keys"
Oct 30 22:58:47.307488 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.307465    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.332286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.330003    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:58:47.332286 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.330950    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:58:47.334420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.334401    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:58:47.334721 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.334706    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:58:47.347551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.347501    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"]
Oct 30 22:58:47.367846 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.367647    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.369094 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.368999    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.369331 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.369099    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg752\" (UniqueName: \"kubernetes.io/projected/88cda865-1fb3-476f-be8c-8963729cf873-kube-api-access-pg752\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.369467 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.369432    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.378330 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.378299    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"]
Oct 30 22:58:47.387650 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.387622    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"]
Oct 30 22:58:47.471804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.471753    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.472009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.471911    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.472009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.471974    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.472137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.472096    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.472137 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.472131    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjsc\" (UniqueName: \"kubernetes.io/projected/1099cf6e-bf69-4056-a065-4024c33622df-kube-api-access-shjsc\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.472294 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.472209    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg752\" (UniqueName: \"kubernetes.io/projected/88cda865-1fb3-476f-be8c-8963729cf873-kube-api-access-pg752\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.473138 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.473113    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.475158 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.475127    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.492105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.492070    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg752\" (UniqueName: \"kubernetes.io/projected/88cda865-1fb3-476f-be8c-8963729cf873-kube-api-access-pg752\") pod \"maas-api-revoke-sub-a8d4ba70-9sb77\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.573862 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.573774    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.574063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.573891    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.574063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.574040    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjsc\" (UniqueName: \"kubernetes.io/projected/1099cf6e-bf69-4056-a065-4024c33622df-kube-api-access-shjsc\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.574937 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.574904    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.576575 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.576543    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.592610 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.592579    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjsc\" (UniqueName: \"kubernetes.io/projected/1099cf6e-bf69-4056-a065-4024c33622df-kube-api-access-shjsc\") pod \"maas-api-revoke-sub-f48f25e1-rh5sj\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.640960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.640924    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:47.703296 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.702475    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:47.972147 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:47.972110    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"]
Oct 30 22:58:47.974339 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:58:47.974313    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88cda865_1fb3_476f_be8c_8963729cf873.slice/crio-45c01a8c9f94ea9cdd24ee6eab06ffb9610c875162f3466694812282ae585268 WatchSource:0}: Error finding container 45c01a8c9f94ea9cdd24ee6eab06ffb9610c875162f3466694812282ae585268: Status 404 returned error can't find the container with id 45c01a8c9f94ea9cdd24ee6eab06ffb9610c875162f3466694812282ae585268
Oct 30 22:58:48.055698 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:58:48.055659    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1099cf6e_bf69_4056_a065_4024c33622df.slice/crio-0645fc136bc884b0d6a8fe8f9efafecbb0840b2c6973240c323a5f925662b100 WatchSource:0}: Error finding container 0645fc136bc884b0d6a8fe8f9efafecbb0840b2c6973240c323a5f925662b100: Status 404 returned error can't find the container with id 0645fc136bc884b0d6a8fe8f9efafecbb0840b2c6973240c323a5f925662b100
Oct 30 22:58:48.058023 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:48.058000    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"]
Oct 30 22:58:48.601243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:48.590340    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp" podUID="9f048de9-6d24-4561-bfcd-d0501b41bbf2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:58:48.601243 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:48.590925    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz" podUID="879ed01f-aaea-42f9-86dd-eb467e927802" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:58:48.728310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:48.723485    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77" event={"ID":"88cda865-1fb3-476f-be8c-8963729cf873","Type":"ContainerStarted","Data":"193bdff4feab1f9986506386dc03947133b41ac64589a365e8f3b41aca42e0e3"}
Oct 30 22:58:48.728310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:48.723569    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77" event={"ID":"88cda865-1fb3-476f-be8c-8963729cf873","Type":"ContainerStarted","Data":"45c01a8c9f94ea9cdd24ee6eab06ffb9610c875162f3466694812282ae585268"}
Oct 30 22:58:48.728310 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:48.726158    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj" event={"ID":"1099cf6e-bf69-4056-a065-4024c33622df","Type":"ContainerStarted","Data":"0645fc136bc884b0d6a8fe8f9efafecbb0840b2c6973240c323a5f925662b100"}
Oct 30 22:58:49.763245 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.761137    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77" podStartSLOduration=2.7611138889999998 podStartE2EDuration="2.761113889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:48.793389063 +0000 UTC m=+3112.832327950" watchObservedRunningTime="2026-10-30 22:58:49.761113889 +0000 UTC m=+3113.800052776"
Oct 30 22:58:49.818256 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.816492    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"]
Oct 30 22:58:49.854377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.847485    4581 generic.go:356] "Generic (PLEG): container finished" podID="88cda865-1fb3-476f-be8c-8963729cf873" containerID="193bdff4feab1f9986506386dc03947133b41ac64589a365e8f3b41aca42e0e3" exitCode=0
Oct 30 22:58:49.854377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.847577    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:49.854377 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.847477    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77" event={"ID":"88cda865-1fb3-476f-be8c-8963729cf873","Type":"ContainerDied","Data":"193bdff4feab1f9986506386dc03947133b41ac64589a365e8f3b41aca42e0e3"}
Oct 30 22:58:49.892354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.856767    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b177e235-e812-4385-aada-80fc0e4874c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp\" (UID: \"b177e235-e812-4385-aada-80fc0e4874c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:49.892354 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.857172    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7qt\" (UniqueName: \"kubernetes.io/projected/b177e235-e812-4385-aada-80fc0e4874c4-kube-api-access-2d7qt\") pod \"payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp\" (UID: \"b177e235-e812-4385-aada-80fc0e4874c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:49.901548 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.895497    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f88979\""
Oct 30 22:58:49.904067 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.904040    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f88979-dockercfg-p2pld\""
Oct 30 22:58:49.928598 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.928529    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"]
Oct 30 22:58:49.937079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.931426    4581 generic.go:356] "Generic (PLEG): container finished" podID="1099cf6e-bf69-4056-a065-4024c33622df" containerID="c5d87dad5f0dd57ee498892e4a3e052d020a7f760af87b3ba90364df81aa9a6f" exitCode=0
Oct 30 22:58:49.937079 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.931470    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj" event={"ID":"1099cf6e-bf69-4056-a065-4024c33622df","Type":"ContainerDied","Data":"c5d87dad5f0dd57ee498892e4a3e052d020a7f760af87b3ba90364df81aa9a6f"}
Oct 30 22:58:49.938344 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.938315    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"]
Oct 30 22:58:49.956505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.952191    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:49.956505 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.952437    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"]
Oct 30 22:58:49.959099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.959069    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b177e235-e812-4385-aada-80fc0e4874c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp\" (UID: \"b177e235-e812-4385-aada-80fc0e4874c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:49.959486 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.959461    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550908fa-315e-4014-88c1-075b25a56178-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j\" (UID: \"550908fa-315e-4014-88c1-075b25a56178\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:49.959631 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.959613    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5c7\" (UniqueName: \"kubernetes.io/projected/550908fa-315e-4014-88c1-075b25a56178-kube-api-access-rl5c7\") pod \"payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j\" (UID: \"550908fa-315e-4014-88c1-075b25a56178\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:49.959960 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.959939    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7qt\" (UniqueName: \"kubernetes.io/projected/b177e235-e812-4385-aada-80fc0e4874c4-kube-api-access-2d7qt\") pod \"payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp\" (UID: \"b177e235-e812-4385-aada-80fc0e4874c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:49.960203 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.960177    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b177e235-e812-4385-aada-80fc0e4874c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp\" (UID: \"b177e235-e812-4385-aada-80fc0e4874c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:49.991291 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:49.990182    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7qt\" (UniqueName: \"kubernetes.io/projected/b177e235-e812-4385-aada-80fc0e4874c4-kube-api-access-2d7qt\") pod \"payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp\" (UID: \"b177e235-e812-4385-aada-80fc0e4874c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:50.063887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.062414    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550908fa-315e-4014-88c1-075b25a56178-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j\" (UID: \"550908fa-315e-4014-88c1-075b25a56178\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:50.063887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.062462    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5c7\" (UniqueName: \"kubernetes.io/projected/550908fa-315e-4014-88c1-075b25a56178-kube-api-access-rl5c7\") pod \"payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j\" (UID: \"550908fa-315e-4014-88c1-075b25a56178\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:50.063887 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.063767    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550908fa-315e-4014-88c1-075b25a56178-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j\" (UID: \"550908fa-315e-4014-88c1-075b25a56178\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:50.081107 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.081080    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:50.122298 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.122011    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5c7\" (UniqueName: \"kubernetes.io/projected/550908fa-315e-4014-88c1-075b25a56178-kube-api-access-rl5c7\") pod \"payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j\" (UID: \"550908fa-315e-4014-88c1-075b25a56178\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:50.124091 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.123903    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:50.174252 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.165500    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:58:50.306355 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.306325    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:58:50.371941 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.371786    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879ed01f-aaea-42f9-86dd-eb467e927802-kube-api-access-29m5l\" (UniqueName: \"kubernetes.io/projected/879ed01f-aaea-42f9-86dd-eb467e927802-kube-api-access-29m5l\") pod \"879ed01f-aaea-42f9-86dd-eb467e927802\" (UID: \"879ed01f-aaea-42f9-86dd-eb467e927802\") "
Oct 30 22:58:50.372465 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.372404    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879ed01f-aaea-42f9-86dd-eb467e927802-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879ed01f-aaea-42f9-86dd-eb467e927802-plugins-config-volume\") pod \"879ed01f-aaea-42f9-86dd-eb467e927802\" (UID: \"879ed01f-aaea-42f9-86dd-eb467e927802\") "
Oct 30 22:58:50.373995 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.373969    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879ed01f-aaea-42f9-86dd-eb467e927802-plugins-config-volume" pod "879ed01f-aaea-42f9-86dd-eb467e927802" (UID: "879ed01f-aaea-42f9-86dd-eb467e927802"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:50.389514 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.389481    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879ed01f-aaea-42f9-86dd-eb467e927802-kube-api-access-29m5l" pod "879ed01f-aaea-42f9-86dd-eb467e927802" (UID: "879ed01f-aaea-42f9-86dd-eb467e927802"). InnerVolumeSpecName "kube-api-access-29m5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:50.474437 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.474390    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f048de9-6d24-4561-bfcd-d0501b41bbf2-kube-api-access-rtjjh\" (UniqueName: \"kubernetes.io/projected/9f048de9-6d24-4561-bfcd-d0501b41bbf2-kube-api-access-rtjjh\") pod \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\" (UID: \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\") "
Oct 30 22:58:50.474608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.474578    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f048de9-6d24-4561-bfcd-d0501b41bbf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f048de9-6d24-4561-bfcd-d0501b41bbf2-plugins-config-volume\") pod \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\" (UID: \"9f048de9-6d24-4561-bfcd-d0501b41bbf2\") "
Oct 30 22:58:50.474946 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.474916    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f048de9-6d24-4561-bfcd-d0501b41bbf2-plugins-config-volume" pod "9f048de9-6d24-4561-bfcd-d0501b41bbf2" (UID: "9f048de9-6d24-4561-bfcd-d0501b41bbf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:50.475199 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.475183    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879ed01f-aaea-42f9-86dd-eb467e927802-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:50.475281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.475204    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f048de9-6d24-4561-bfcd-d0501b41bbf2-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:50.475281 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.475236    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29m5l\" (UniqueName: \"kubernetes.io/projected/879ed01f-aaea-42f9-86dd-eb467e927802-kube-api-access-29m5l\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:50.476617 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.476595    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f048de9-6d24-4561-bfcd-d0501b41bbf2-kube-api-access-rtjjh" pod "9f048de9-6d24-4561-bfcd-d0501b41bbf2" (UID: "9f048de9-6d24-4561-bfcd-d0501b41bbf2"). InnerVolumeSpecName "kube-api-access-rtjjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:50.501339 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.501310    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"]
Oct 30 22:58:50.501837 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:58:50.501808    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb177e235_e812_4385_aada_80fc0e4874c4.slice/crio-eea90e6b8dfbcfd9d90657ff1fec669fdbd81d6edd0df5d3238c345f1c09b209 WatchSource:0}: Error finding container eea90e6b8dfbcfd9d90657ff1fec669fdbd81d6edd0df5d3238c345f1c09b209: Status 404 returned error can't find the container with id eea90e6b8dfbcfd9d90657ff1fec669fdbd81d6edd0df5d3238c345f1c09b209
Oct 30 22:58:50.558517 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.558489    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"]
Oct 30 22:58:50.563717 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:58:50.563687    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550908fa_315e_4014_88c1_075b25a56178.slice/crio-73bd64bb3b5240983b459488f252e6a7b2513b78cb1e59374757db7a39a6846b WatchSource:0}: Error finding container 73bd64bb3b5240983b459488f252e6a7b2513b78cb1e59374757db7a39a6846b: Status 404 returned error can't find the container with id 73bd64bb3b5240983b459488f252e6a7b2513b78cb1e59374757db7a39a6846b
Oct 30 22:58:50.576949 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.576918    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtjjh\" (UniqueName: \"kubernetes.io/projected/9f048de9-6d24-4561-bfcd-d0501b41bbf2-kube-api-access-rtjjh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:50.948773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.948668    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j" event={"ID":"550908fa-315e-4014-88c1-075b25a56178","Type":"ContainerStarted","Data":"7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4"}
Oct 30 22:58:50.948773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.948734    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j" event={"ID":"550908fa-315e-4014-88c1-075b25a56178","Type":"ContainerStarted","Data":"73bd64bb3b5240983b459488f252e6a7b2513b78cb1e59374757db7a39a6846b"}
Oct 30 22:58:50.948773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.948783    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:58:50.950577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.950549    4581 generic.go:356] "Generic (PLEG): container finished" podID="879ed01f-aaea-42f9-86dd-eb467e927802" containerID="80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff" exitCode=0
Oct 30 22:58:50.950708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.950622    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"
Oct 30 22:58:50.950708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.950629    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz" event={"ID":"879ed01f-aaea-42f9-86dd-eb467e927802","Type":"ContainerDied","Data":"80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff"}
Oct 30 22:58:50.950708 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.950671    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz" event={"ID":"879ed01f-aaea-42f9-86dd-eb467e927802","Type":"ContainerDied","Data":"17ff64dfac8c3468296ac44bad157377e96c7cfc8e7fa26c9ce343bd70a727e0"}
Oct 30 22:58:50.950885 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.950709    4581 scope.go:122] "RemoveContainer" containerID="80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff"
Oct 30 22:58:50.953948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.953926    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp" event={"ID":"b177e235-e812-4385-aada-80fc0e4874c4","Type":"ContainerStarted","Data":"954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2"}
Oct 30 22:58:50.954114 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.954101    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp" event={"ID":"b177e235-e812-4385-aada-80fc0e4874c4","Type":"ContainerStarted","Data":"eea90e6b8dfbcfd9d90657ff1fec669fdbd81d6edd0df5d3238c345f1c09b209"}
Oct 30 22:58:50.954822 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.954800    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:58:50.958131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.956270    4581 generic.go:356] "Generic (PLEG): container finished" podID="9f048de9-6d24-4561-bfcd-d0501b41bbf2" containerID="b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407" exitCode=0
Oct 30 22:58:50.958131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.956536    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"
Oct 30 22:58:50.958131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.956565    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp" event={"ID":"9f048de9-6d24-4561-bfcd-d0501b41bbf2","Type":"ContainerDied","Data":"b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407"}
Oct 30 22:58:50.958131 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.956622    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp" event={"ID":"9f048de9-6d24-4561-bfcd-d0501b41bbf2","Type":"ContainerDied","Data":"e6d2858de1a4b231356686a7d9d7c078a3e9c29f49b266bac7e1a93c708dc796"}
Oct 30 22:58:50.968502 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.968484    4581 scope.go:122] "RemoveContainer" containerID="80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff"
Oct 30 22:58:50.968986 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:58:50.968956    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff\": container with ID starting with 80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff not found: ID does not exist" containerID="80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff"
Oct 30 22:58:50.969087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.968997    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff"} err="failed to get container status \"80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff\": rpc error: code = NotFound desc = could not find container \"80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff\": container with ID starting with 80d11a913522600ba980325e49e7e58c2a54b95c455ce4bc6a85c7450bcf94ff not found: ID does not exist"
Oct 30 22:58:50.969087 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.969020    4581 scope.go:122] "RemoveContainer" containerID="b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407"
Oct 30 22:58:50.988600 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.988578    4581 scope.go:122] "RemoveContainer" containerID="b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407"
Oct 30 22:58:50.988938 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:58:50.988916    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407\": container with ID starting with b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407 not found: ID does not exist" containerID="b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407"
Oct 30 22:58:50.989016 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.988949    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407"} err="failed to get container status \"b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407\": rpc error: code = NotFound desc = could not find container \"b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407\": container with ID starting with b3cfc10d37baa3f0d1029393446d5714c70327dac49277b19ad4d8be9d5f3407 not found: ID does not exist"
Oct 30 22:58:50.996669 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:50.996619    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j" podStartSLOduration=1.9966048939999999 podStartE2EDuration="1.996604894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:50.994966187 +0000 UTC m=+3115.033905073" watchObservedRunningTime="2026-10-30 22:58:50.996604894 +0000 UTC m=+3115.035543799"
Oct 30 22:58:51.047372 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.047344    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"]
Oct 30 22:58:51.057723 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.057678    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0ad7c5-dffdb7775-kt9zp"]
Oct 30 22:58:51.097272 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.091659    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp" podStartSLOduration=2.091638483 podStartE2EDuration="2.091638483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:51.084121948 +0000 UTC m=+3115.123060835" watchObservedRunningTime="2026-10-30 22:58:51.091638483 +0000 UTC m=+3115.130577371"
Oct 30 22:58:51.145966 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.145931    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"]
Oct 30 22:58:51.154371 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.154337    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0ad7c5-7b6cc5f5c5-8c2qz"]
Oct 30 22:58:51.226317 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.226191    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:51.266327 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.265272    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:51.294577 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.294545    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-openshift-ca\") pod \"88cda865-1fb3-476f-be8c-8963729cf873\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") "
Oct 30 22:58:51.294775 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.294684    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-xks-ca\") pod \"88cda865-1fb3-476f-be8c-8963729cf873\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") "
Oct 30 22:58:51.294888 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.294869    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-xks-ca\") pod \"1099cf6e-bf69-4056-a065-4024c33622df\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") "
Oct 30 22:58:51.295111 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.295089    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88cda865-1fb3-476f-be8c-8963729cf873-kube-api-access-pg752\" (UniqueName: \"kubernetes.io/projected/88cda865-1fb3-476f-be8c-8963729cf873-kube-api-access-pg752\") pod \"88cda865-1fb3-476f-be8c-8963729cf873\" (UID: \"88cda865-1fb3-476f-be8c-8963729cf873\") "
Oct 30 22:58:51.295356 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.295337    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1099cf6e-bf69-4056-a065-4024c33622df-kube-api-access-shjsc\" (UniqueName: \"kubernetes.io/projected/1099cf6e-bf69-4056-a065-4024c33622df-kube-api-access-shjsc\") pod \"1099cf6e-bf69-4056-a065-4024c33622df\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") "
Oct 30 22:58:51.295491 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.295476    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-openshift-ca\") pod \"1099cf6e-bf69-4056-a065-4024c33622df\" (UID: \"1099cf6e-bf69-4056-a065-4024c33622df\") "
Oct 30 22:58:51.296481 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.295362    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-openshift-ca" pod "88cda865-1fb3-476f-be8c-8963729cf873" (UID: "88cda865-1fb3-476f-be8c-8963729cf873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:51.297089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.297046    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-openshift-ca" pod "1099cf6e-bf69-4056-a065-4024c33622df" (UID: "1099cf6e-bf69-4056-a065-4024c33622df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:51.301280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.300691    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-xks-ca" pod "1099cf6e-bf69-4056-a065-4024c33622df" (UID: "1099cf6e-bf69-4056-a065-4024c33622df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:51.301280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.300733    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cda865-1fb3-476f-be8c-8963729cf873-kube-api-access-pg752" pod "88cda865-1fb3-476f-be8c-8963729cf873" (UID: "88cda865-1fb3-476f-be8c-8963729cf873"). InnerVolumeSpecName "kube-api-access-pg752". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:51.301280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.300804    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1099cf6e-bf69-4056-a065-4024c33622df-kube-api-access-shjsc" pod "1099cf6e-bf69-4056-a065-4024c33622df" (UID: "1099cf6e-bf69-4056-a065-4024c33622df"). InnerVolumeSpecName "kube-api-access-shjsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:51.301280 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.301072    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-xks-ca" pod "88cda865-1fb3-476f-be8c-8963729cf873" (UID: "88cda865-1fb3-476f-be8c-8963729cf873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:51.398068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.398031    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.398068 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.398069    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg752\" (UniqueName: \"kubernetes.io/projected/88cda865-1fb3-476f-be8c-8963729cf873-kube-api-access-pg752\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.398342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.398086    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shjsc\" (UniqueName: \"kubernetes.io/projected/1099cf6e-bf69-4056-a065-4024c33622df-kube-api-access-shjsc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.398342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.398103    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1099cf6e-bf69-4056-a065-4024c33622df-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.398342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.398117    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.398342 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.398133    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88cda865-1fb3-476f-be8c-8963729cf873-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:58:51.968409 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.968377    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"
Oct 30 22:58:51.968909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.968409    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj" event={"ID":"1099cf6e-bf69-4056-a065-4024c33622df","Type":"ContainerDied","Data":"0645fc136bc884b0d6a8fe8f9efafecbb0840b2c6973240c323a5f925662b100"}
Oct 30 22:58:51.968909 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.968463    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0645fc136bc884b0d6a8fe8f9efafecbb0840b2c6973240c323a5f925662b100"
Oct 30 22:58:51.972500 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.972477    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"
Oct 30 22:58:51.972656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.972471    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77" event={"ID":"88cda865-1fb3-476f-be8c-8963729cf873","Type":"ContainerDied","Data":"45c01a8c9f94ea9cdd24ee6eab06ffb9610c875162f3466694812282ae585268"}
Oct 30 22:58:51.972656 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:51.972613    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c01a8c9f94ea9cdd24ee6eab06ffb9610c875162f3466694812282ae585268"
Oct 30 22:58:52.099760 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:52.099719    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879ed01f-aaea-42f9-86dd-eb467e927802" path="/var/lib/kubelet/pods/879ed01f-aaea-42f9-86dd-eb467e927802/volumes"
Oct 30 22:58:52.100325 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:52.100302    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f048de9-6d24-4561-bfcd-d0501b41bbf2" path="/var/lib/kubelet/pods/9f048de9-6d24-4561-bfcd-d0501b41bbf2/volumes"
Oct 30 22:58:52.512948 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:52.512909    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"]
Oct 30 22:58:59.557140 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557101    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"]
Oct 30 22:58:59.557873 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557857    4581 cpu_manager.go:414] "containerMap: removing container" podUID="88cda865-1fb3-476f-be8c-8963729cf873" containerName="revoke-keys"
Oct 30 22:58:59.557920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557874    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88cda865-1fb3-476f-be8c-8963729cf873" containerName="revoke-keys"
Oct 30 22:58:59.557920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557892    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1099cf6e-bf69-4056-a065-4024c33622df" containerName="revoke-keys"
Oct 30 22:58:59.557920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557897    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1099cf6e-bf69-4056-a065-4024c33622df" containerName="revoke-keys"
Oct 30 22:58:59.557920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557906    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9f048de9-6d24-4561-bfcd-d0501b41bbf2" containerName="payload-pre-processing"
Oct 30 22:58:59.557920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557910    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f048de9-6d24-4561-bfcd-d0501b41bbf2" containerName="payload-pre-processing"
Oct 30 22:58:59.557920 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557919    4581 cpu_manager.go:414] "containerMap: removing container" podUID="879ed01f-aaea-42f9-86dd-eb467e927802" containerName="payload-processing"
Oct 30 22:58:59.558105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.557926    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879ed01f-aaea-42f9-86dd-eb467e927802" containerName="payload-processing"
Oct 30 22:58:59.558105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.558036    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="88cda865-1fb3-476f-be8c-8963729cf873" containerName="revoke-keys"
Oct 30 22:58:59.558105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.558047    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1099cf6e-bf69-4056-a065-4024c33622df" containerName="revoke-keys"
Oct 30 22:58:59.558105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.558054    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f048de9-6d24-4561-bfcd-d0501b41bbf2" containerName="payload-pre-processing"
Oct 30 22:58:59.558105 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.558061    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="879ed01f-aaea-42f9-86dd-eb467e927802" containerName="payload-processing"
Oct 30 22:58:59.562031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.562008    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:58:59.584616 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.584587    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"]
Oct 30 22:58:59.588847 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.587029    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f94561-dockercfg-xm6r4\""
Oct 30 22:58:59.602955 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.602893    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f94561\""
Oct 30 22:58:59.697244 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.697185    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn\" (UID: \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:58:59.697493 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.697455    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8zf\" (UniqueName: \"kubernetes.io/projected/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-kube-api-access-vd8zf\") pod \"payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn\" (UID: \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:58:59.824365 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.824205    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn\" (UID: \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:58:59.824818 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.824794    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8zf\" (UniqueName: \"kubernetes.io/projected/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-kube-api-access-vd8zf\") pod \"payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn\" (UID: \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:58:59.826267 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.825864    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn\" (UID: \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:58:59.866164 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.866131    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8zf\" (UniqueName: \"kubernetes.io/projected/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-kube-api-access-vd8zf\") pod \"payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn\" (UID: \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:58:59.880458 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:58:59.880428    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:59:00.215806 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:00.215775    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"]
Oct 30 22:59:00.217115 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:59:00.217090    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6fed26_0cef_4d4a_9d3e_76643a5215dd.slice/crio-9ccdc8f571cd0a53e34be1d6119f737b0c25fc5f2caefb4b030ef551092ee941 WatchSource:0}: Error finding container 9ccdc8f571cd0a53e34be1d6119f737b0c25fc5f2caefb4b030ef551092ee941: Status 404 returned error can't find the container with id 9ccdc8f571cd0a53e34be1d6119f737b0c25fc5f2caefb4b030ef551092ee941
Oct 30 22:59:01.120549 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:01.120513    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn" event={"ID":"ea6fed26-0cef-4d4a-9d3e-76643a5215dd","Type":"ContainerStarted","Data":"460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453"}
Oct 30 22:59:01.121031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:01.120603    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn" event={"ID":"ea6fed26-0cef-4d4a-9d3e-76643a5215dd","Type":"ContainerStarted","Data":"9ccdc8f571cd0a53e34be1d6119f737b0c25fc5f2caefb4b030ef551092ee941"}
Oct 30 22:59:01.121031 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:01.120626    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:59:01.161793 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:01.161733    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn" podStartSLOduration=2.161715872 podStartE2EDuration="2.161715872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:01.160131633 +0000 UTC m=+3125.199070523" watchObservedRunningTime="2026-10-30 22:59:01.161715872 +0000 UTC m=+3125.200654760"
Oct 30 22:59:01.976148 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:01.976118    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 22:59:01.976324 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:01.976276    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 22:59:07.101273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.101232    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"]
Oct 30 22:59:07.107439 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.107417    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.120545 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.120508    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cdfa86-openshift-default-dockercfg-x95jg\""
Oct 30 22:59:07.138329 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.138293    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"]
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.335598    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59h8s\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-kube-api-access-59h8s\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.335744    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istiod-ca-cert\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.335894    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-envoy\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.335973    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-certs\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.336003    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-credential-socket\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.336043    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-podinfo\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.336067    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-data\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.336107    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-socket\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.336322 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.336191    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-token\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438271 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438235    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istiod-ca-cert\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438286    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-envoy\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438333    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-certs\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438350    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-credential-socket\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438378    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-podinfo\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438399    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-data\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438464 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438428    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-socket\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438500    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-token\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438792 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438655    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59h8s\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-kube-api-access-59h8s\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438902 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438876    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-credential-socket\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.438965 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.438934    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-certs\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.439375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.439093    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-data\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.439375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.439257    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-socket\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.439375 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.439306    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istiod-ca-cert\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.441141 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.441118    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-envoy\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.441494 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.441472    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-podinfo\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.457157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.457125    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-token\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.457479 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.457457    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59h8s\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-kube-api-access-59h8s\") pod \"e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") " pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.536772 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.536724    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv" podUID="c4b465d7-ada9-4894-aa25-613cf1145ecf" containerName="main" containerID="cri-o://21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47" gracePeriod=45
Oct 30 22:59:07.723249 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.723136    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:07.839295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.837151    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:59:07.979309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.978210    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b465d7-ada9-4894-aa25-613cf1145ecf-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b465d7-ada9-4894-aa25-613cf1145ecf-tls-certs\") pod \"c4b465d7-ada9-4894-aa25-613cf1145ecf\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") "
Oct 30 22:59:07.979309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.978330    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-tmp-dir\") pod \"c4b465d7-ada9-4894-aa25-613cf1145ecf\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") "
Oct 30 22:59:07.979309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.978426    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-home\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-home\") pod \"c4b465d7-ada9-4894-aa25-613cf1145ecf\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") "
Oct 30 22:59:07.979309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.978503    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-dshm\") pod \"c4b465d7-ada9-4894-aa25-613cf1145ecf\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") "
Oct 30 22:59:07.979309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.978598    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-model-cache\") pod \"c4b465d7-ada9-4894-aa25-613cf1145ecf\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") "
Oct 30 22:59:07.979309 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.978636    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b465d7-ada9-4894-aa25-613cf1145ecf-kube-api-access-7bp9f\" (UniqueName: \"kubernetes.io/projected/c4b465d7-ada9-4894-aa25-613cf1145ecf-kube-api-access-7bp9f\") pod \"c4b465d7-ada9-4894-aa25-613cf1145ecf\" (UID: \"c4b465d7-ada9-4894-aa25-613cf1145ecf\") "
Oct 30 22:59:07.997290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.987326    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-home" pod "c4b465d7-ada9-4894-aa25-613cf1145ecf" (UID: "c4b465d7-ada9-4894-aa25-613cf1145ecf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:07.997290 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:07.992313    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-tmp-dir" pod "c4b465d7-ada9-4894-aa25-613cf1145ecf" (UID: "c4b465d7-ada9-4894-aa25-613cf1145ecf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:08.004729 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.004692    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-model-cache" pod "c4b465d7-ada9-4894-aa25-613cf1145ecf" (UID: "c4b465d7-ada9-4894-aa25-613cf1145ecf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:08.081195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.081078    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:08.081195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.081122    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:08.081195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.081136    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:08.145983 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.144582    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b465d7-ada9-4894-aa25-613cf1145ecf-tls-certs" pod "c4b465d7-ada9-4894-aa25-613cf1145ecf" (UID: "c4b465d7-ada9-4894-aa25-613cf1145ecf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:08.150417 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.148419    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b465d7-ada9-4894-aa25-613cf1145ecf-kube-api-access-7bp9f" pod "c4b465d7-ada9-4894-aa25-613cf1145ecf" (UID: "c4b465d7-ada9-4894-aa25-613cf1145ecf"). InnerVolumeSpecName "kube-api-access-7bp9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:08.159891 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.156092    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-dshm" pod "c4b465d7-ada9-4894-aa25-613cf1145ecf" (UID: "c4b465d7-ada9-4894-aa25-613cf1145ecf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:08.167273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.162544    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:59:08.167273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.162643    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:59:08.167273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.162711    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 22:59:08.167273 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.166614    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"]
Oct 30 22:59:08.183332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.183307    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b465d7-ada9-4894-aa25-613cf1145ecf-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:08.183332 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.183334    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b465d7-ada9-4894-aa25-613cf1145ecf-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:08.183498 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.183347    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bp9f\" (UniqueName: \"kubernetes.io/projected/c4b465d7-ada9-4894-aa25-613cf1145ecf-kube-api-access-7bp9f\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:08.421178 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.421143    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" event={"ID":"c069b4a9-3dd5-4144-a15e-e3df78c5f554","Type":"ContainerStarted","Data":"a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c"}
Oct 30 22:59:08.421400 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.421205    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" event={"ID":"c069b4a9-3dd5-4144-a15e-e3df78c5f554","Type":"ContainerStarted","Data":"10b1181aa274a156f3de90a390846ad5a9dc598d8ded3e0b81b8917a7315f36e"}
Oct 30 22:59:08.423632 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.423604    4581 generic.go:356] "Generic (PLEG): container finished" podID="c4b465d7-ada9-4894-aa25-613cf1145ecf" containerID="21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47" exitCode=0
Oct 30 22:59:08.423773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.423728    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv" event={"ID":"c4b465d7-ada9-4894-aa25-613cf1145ecf","Type":"ContainerDied","Data":"21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47"}
Oct 30 22:59:08.423773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.423761    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv" event={"ID":"c4b465d7-ada9-4894-aa25-613cf1145ecf","Type":"ContainerDied","Data":"d0a395ed94ca947d1807a8cd94fc3e46efa7fa426eb739b4469adcf2d928828d"}
Oct 30 22:59:08.423880 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.423781    4581 scope.go:122] "RemoveContainer" containerID="21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47"
Oct 30 22:59:08.423970 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.423955    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"
Oct 30 22:59:08.439266 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.439244    4581 scope.go:122] "RemoveContainer" containerID="21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47"
Oct 30 22:59:08.439607 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:59:08.439585    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47\": container with ID starting with 21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47 not found: ID does not exist" containerID="21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47"
Oct 30 22:59:08.439705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.439613    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47"} err="failed to get container status \"21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47\": rpc error: code = NotFound desc = could not find container \"21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47\": container with ID starting with 21a4cdd8228ea351faf451ac23d169aef7cc32b3b76abacc0e8b43c5d7936d47 not found: ID does not exist"
Oct 30 22:59:08.529260 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.529116    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" podStartSLOduration=1.5290905380000002 podStartE2EDuration="1.529090538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:08.471516058 +0000 UTC m=+3132.510454945" watchObservedRunningTime="2026-10-30 22:59:08.529090538 +0000 UTC m=+3132.568029428"
Oct 30 22:59:08.529453 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.529436    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"]
Oct 30 22:59:08.538099 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.538065    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e14e00/rate-test-model-e14e00-kserve-6dc9d546d9-4p2hv"]
Oct 30 22:59:08.724097 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:08.724053    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:09.730700 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:09.730666    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:10.098165 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:10.098073    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b465d7-ada9-4894-aa25-613cf1145ecf" path="/var/lib/kubelet/pods/c4b465d7-ada9-4894-aa25-613cf1145ecf/volumes"
Oct 30 22:59:10.441934 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:10.441901    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:10.443010 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:10.442991    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 22:59:10.820844 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:59:10.820747    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:59:12.134438 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.134410    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 22:59:12.802441 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.802403    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"]
Oct 30 22:59:12.803450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.803426    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c4b465d7-ada9-4894-aa25-613cf1145ecf" containerName="main"
Oct 30 22:59:12.803450 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.803449    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b465d7-ada9-4894-aa25-613cf1145ecf" containerName="main"
Oct 30 22:59:12.803707 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.803692    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b465d7-ada9-4894-aa25-613cf1145ecf" containerName="main"
Oct 30 22:59:12.809015 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.808991    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:12.829945 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.829915    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"]
Oct 30 22:59:12.872074 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.872033    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs\" (UID: \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:12.872398 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.872376    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8cr\" (UniqueName: \"kubernetes.io/projected/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-kube-api-access-rn8cr\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs\" (UID: \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:12.919563 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.919528    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cdfa86\""
Oct 30 22:59:12.919938 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.919919    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cdfa86-dockercfg-7shml\""
Oct 30 22:59:12.975313 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.975275    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8cr\" (UniqueName: \"kubernetes.io/projected/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-kube-api-access-rn8cr\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs\" (UID: \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:12.995250 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.976333    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs\" (UID: \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:12.998115 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:12.996427    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs\" (UID: \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:13.031009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:13.030972    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8cr\" (UniqueName: \"kubernetes.io/projected/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-kube-api-access-rn8cr\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs\" (UID: \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:13.132009 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:13.131972    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:13.453468 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:13.453436    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"]
Oct 30 22:59:13.454619 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:59:13.454586    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62f335e_cd92_45d1_96b9_ba5e7099cf8e.slice/crio-fb5129163c5f86db4b8663958d002a8fdb2b57a23519899c74b50044232c8978 WatchSource:0}: Error finding container fb5129163c5f86db4b8663958d002a8fdb2b57a23519899c74b50044232c8978: Status 404 returned error can't find the container with id fb5129163c5f86db4b8663958d002a8fdb2b57a23519899c74b50044232c8978
Oct 30 22:59:13.472942 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:13.472916    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs" event={"ID":"c62f335e-cd92-45d1-96b9-ba5e7099cf8e","Type":"ContainerStarted","Data":"fb5129163c5f86db4b8663958d002a8fdb2b57a23519899c74b50044232c8978"}
Oct 30 22:59:14.496045 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:14.495999    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs" event={"ID":"c62f335e-cd92-45d1-96b9-ba5e7099cf8e","Type":"ContainerStarted","Data":"54d240d02b1b282af3ac18d7fa3cf67492a22c41f630cb03a77c40c0236e2626"}
Oct 30 22:59:14.496480 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:14.496107    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:14.531383 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:14.531331    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs" podStartSLOduration=2.531317923 podStartE2EDuration="2.531317923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:14.529665735 +0000 UTC m=+3138.568604621" watchObservedRunningTime="2026-10-30 22:59:14.531317923 +0000 UTC m=+3138.570256809"
Oct 30 22:59:22.562804 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.562767    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"]
Oct 30 22:59:22.569151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.569125    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.589063 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.589032    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"]
Oct 30 22:59:22.589586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.589561    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:59:22.589586 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.589581    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:59:22.589812 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.589691    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:59:22.590832 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.590811    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:59:22.655055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.655019    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.655304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.655152    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.655304 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.655211    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/b7f2c635-a735-44fa-980e-9ec696ff8386-kube-api-access-dlh5s\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.757169 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.757137    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.757418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.757264    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.757418 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.757338    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/b7f2c635-a735-44fa-980e-9ec696ff8386-kube-api-access-dlh5s\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.758405 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.758373    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.760319 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.760291    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.775279 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.775251    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/b7f2c635-a735-44fa-980e-9ec696ff8386-kube-api-access-dlh5s\") pod \"maas-api-revoke-sub-3fab2ecd-5zz9w\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:22.896335 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:22.893558    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:23.228808 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:23.228781    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"]
Oct 30 22:59:23.229550 ip-10-0-135-132 kubenswrapper[4581]: W1030 22:59:23.229520    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f2c635_a735_44fa_980e_9ec696ff8386.slice/crio-e245bbb3afba351a3c5ec7b5fcd882fce0f02e75b2c040fbc87ae782bf9eb2d2 WatchSource:0}: Error finding container e245bbb3afba351a3c5ec7b5fcd882fce0f02e75b2c040fbc87ae782bf9eb2d2: Status 404 returned error can't find the container with id e245bbb3afba351a3c5ec7b5fcd882fce0f02e75b2c040fbc87ae782bf9eb2d2
Oct 30 22:59:23.584872 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:23.584811    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w" event={"ID":"b7f2c635-a735-44fa-980e-9ec696ff8386","Type":"ContainerStarted","Data":"4fb1a460a939ce5fcc2be392f155c6a20b22653b53e833007c2ade6a96715785"}
Oct 30 22:59:23.584872 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:23.584875    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w" event={"ID":"b7f2c635-a735-44fa-980e-9ec696ff8386","Type":"ContainerStarted","Data":"e245bbb3afba351a3c5ec7b5fcd882fce0f02e75b2c040fbc87ae782bf9eb2d2"}
Oct 30 22:59:23.632121 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:23.632065    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w" podStartSLOduration=1.6320503130000001 podStartE2EDuration="1.632050313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:23.621031421 +0000 UTC m=+3147.659970310" watchObservedRunningTime="2026-10-30 22:59:23.632050313 +0000 UTC m=+3147.670989199"
Oct 30 22:59:24.598739 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:24.598700    4581 generic.go:356] "Generic (PLEG): container finished" podID="b7f2c635-a735-44fa-980e-9ec696ff8386" containerID="4fb1a460a939ce5fcc2be392f155c6a20b22653b53e833007c2ade6a96715785" exitCode=0
Oct 30 22:59:24.599157 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:24.598782    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w" event={"ID":"b7f2c635-a735-44fa-980e-9ec696ff8386","Type":"ContainerDied","Data":"4fb1a460a939ce5fcc2be392f155c6a20b22653b53e833007c2ade6a96715785"}
Oct 30 22:59:25.509554 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.509518    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 22:59:25.820027 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.819999    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:25.905969 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.905930    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-openshift-ca\") pod \"b7f2c635-a735-44fa-980e-9ec696ff8386\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") "
Oct 30 22:59:25.906195 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.906053    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-xks-ca\") pod \"b7f2c635-a735-44fa-980e-9ec696ff8386\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") "
Oct 30 22:59:25.906388 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.906345    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f2c635-a735-44fa-980e-9ec696ff8386-kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/b7f2c635-a735-44fa-980e-9ec696ff8386-kube-api-access-dlh5s\") pod \"b7f2c635-a735-44fa-980e-9ec696ff8386\" (UID: \"b7f2c635-a735-44fa-980e-9ec696ff8386\") "
Oct 30 22:59:25.906496 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.906394    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-openshift-ca" pod "b7f2c635-a735-44fa-980e-9ec696ff8386" (UID: "b7f2c635-a735-44fa-980e-9ec696ff8386"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:25.907055 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.907028    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:25.908477 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.908453    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-xks-ca" pod "b7f2c635-a735-44fa-980e-9ec696ff8386" (UID: "b7f2c635-a735-44fa-980e-9ec696ff8386"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:25.909144 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:25.909121    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f2c635-a735-44fa-980e-9ec696ff8386-kube-api-access-dlh5s" pod "b7f2c635-a735-44fa-980e-9ec696ff8386" (UID: "b7f2c635-a735-44fa-980e-9ec696ff8386"). InnerVolumeSpecName "kube-api-access-dlh5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:26.008753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:26.008713    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f2c635-a735-44fa-980e-9ec696ff8386-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:26.008753 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:26.008748    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/b7f2c635-a735-44fa-980e-9ec696ff8386-kube-api-access-dlh5s\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:26.624089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:26.624061    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"
Oct 30 22:59:26.624089 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:26.624064    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w" event={"ID":"b7f2c635-a735-44fa-980e-9ec696ff8386","Type":"ContainerDied","Data":"e245bbb3afba351a3c5ec7b5fcd882fce0f02e75b2c040fbc87ae782bf9eb2d2"}
Oct 30 22:59:26.624395 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:26.624123    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e245bbb3afba351a3c5ec7b5fcd882fce0f02e75b2c040fbc87ae782bf9eb2d2"
Oct 30 22:59:33.310420 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:33.310386    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"]
Oct 30 22:59:33.316295 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:33.316264    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0ad7c5-zfcxp"]
Oct 30 22:59:34.100515 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:34.100475    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b225e238-bde0-41f4-916b-2dbb557c0630" path="/var/lib/kubelet/pods/b225e238-bde0-41f4-916b-2dbb557c0630/volumes"
Oct 30 22:59:40.819015 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:59:40.818962    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 22:59:41.114528 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:41.114493    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"]
Oct 30 22:59:41.115151 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:41.115092    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" containerID="cri-o://1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2" gracePeriod=30
Oct 30 22:59:45.831264 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:45.831176    4581 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:59:45.831673 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:45.831296    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:59:46.446644 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.446615    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:59:46.521046 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521008    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-token\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-token\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521282 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521203    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-credential-socket\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521386 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521368    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-podinfo\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521455 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521429    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-certs\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521551 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521535    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86ad25d-cba7-4736-add7-cc1865fe0feb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86ad25d-cba7-4736-add7-cc1865fe0feb-istiod-ca-cert\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521619 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521597    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-data\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521633    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-socket\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521679 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521667    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-envoy\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521695    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-kube-api-access-nhm5k\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-kube-api-access-nhm5k\") pod \"c86ad25d-cba7-4736-add7-cc1865fe0feb\" (UID: \"c86ad25d-cba7-4736-add7-cc1865fe0feb\") "
Oct 30 22:59:46.521797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521528    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-credential-socket" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:46.521797 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521737    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-certs" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:46.521962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521923    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-socket" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:46.521962 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.521951    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86ad25d-cba7-4736-add7-cc1865fe0feb-istiod-ca-cert" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:46.522206 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.522181    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-data" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:46.522513 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.522493    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.522608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.522544    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.522608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.522555    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86ad25d-cba7-4736-add7-cc1865fe0feb-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.522608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.522562    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.522608 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.522570    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.524597 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.524565    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-envoy" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:46.524890 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.524863    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-kube-api-access-nhm5k" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "kube-api-access-nhm5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:46.525012 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.524989    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-podinfo" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:59:46.525129 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.525066    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-token" pod "c86ad25d-cba7-4736-add7-cc1865fe0feb" (UID: "c86ad25d-cba7-4736-add7-cc1865fe0feb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:46.623705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.623667    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.623705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.623699    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhm5k\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-kube-api-access-nhm5k\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.623705 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.623709    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.623956 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.623717    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86ad25d-cba7-4736-add7-cc1865fe0feb-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 22:59:46.835443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.835356    4581 generic.go:356] "Generic (PLEG): container finished" podID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerID="1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2" exitCode=0
Oct 30 22:59:46.835443 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.835411    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"
Oct 30 22:59:46.835910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.835436    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" event={"ID":"c86ad25d-cba7-4736-add7-cc1865fe0feb","Type":"ContainerDied","Data":"1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2"}
Oct 30 22:59:46.835910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.835491    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5" event={"ID":"c86ad25d-cba7-4736-add7-cc1865fe0feb","Type":"ContainerDied","Data":"10f9dfe91e44b9ab39aea09c68908178a0e764512acbdade43ecb12c6609fc36"}
Oct 30 22:59:46.835910 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.835514    4581 scope.go:122] "RemoveContainer" containerID="1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2"
Oct 30 22:59:46.852773 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.852755    4581 scope.go:122] "RemoveContainer" containerID="1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2"
Oct 30 22:59:46.853099 ip-10-0-135-132 kubenswrapper[4581]: E1030 22:59:46.853080    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2\": container with ID starting with 1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2 not found: ID does not exist" containerID="1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2"
Oct 30 22:59:46.853167 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.853107    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2"} err="failed to get container status \"1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2\": rpc error: code = NotFound desc = could not find container \"1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2\": container with ID starting with 1464e641b5b88e924a6ed3ca4eb4bd35fa0092c816c1aa932cfef043d66a9ab2 not found: ID does not exist"
Oct 30 22:59:46.884171 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.884134    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"]
Oct 30 22:59:46.889189 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:46.889162    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-72ws5"]
Oct 30 22:59:48.100459 ip-10-0-135-132 kubenswrapper[4581]: I1030 22:59:48.100420    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" path="/var/lib/kubelet/pods/c86ad25d-cba7-4736-add7-cc1865fe0feb/volumes"
Oct 30 23:00:00.172096 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.171274    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"]
Oct 30 23:00:00.172845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.172308    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy"
Oct 30 23:00:00.172845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.172327    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy"
Oct 30 23:00:00.172845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.172397    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b7f2c635-a735-44fa-980e-9ec696ff8386" containerName="revoke-keys"
Oct 30 23:00:00.172845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.172405    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f2c635-a735-44fa-980e-9ec696ff8386" containerName="revoke-keys"
Oct 30 23:00:00.176458 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.176424    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f2c635-a735-44fa-980e-9ec696ff8386" containerName="revoke-keys"
Oct 30 23:00:00.176627 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.176469    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86ad25d-cba7-4736-add7-cc1865fe0feb" containerName="istio-proxy"
Oct 30 23:00:00.204353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.204323    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"]
Oct 30 23:00:00.208434 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.208407    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:00.217724 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.216533    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"]
Oct 30 23:00:00.217724 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.217717    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:00.232599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.227937    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"]
Oct 30 23:00:00.232599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.232397    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:00.243261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.233663    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"]
Oct 30 23:00:00.243261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.238196    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:00.243261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.239392    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"]
Oct 30 23:00:00.243261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.242355    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:00.277344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.250664    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:00:00.277344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.251469    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:00:00.277344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.252705    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:00:00.277344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.262052    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"]
Oct 30 23:00:00.277344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.269251    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:00.345282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.332432    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"]
Oct 30 23:00:00.345282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.333791    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:00.345282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.337885    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"]
Oct 30 23:00:00.345282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.338496    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:00.351248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.350388    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"]
Oct 30 23:00:00.360955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.359390    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372339    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nwh\" (UniqueName: \"kubernetes.io/projected/bb8e28dd-5a24-420b-8233-38b38e2bfda3-kube-api-access-m6nwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4\" (UID: \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372394    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4vw\" (UniqueName: \"kubernetes.io/projected/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-kube-api-access-4w4vw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c\" (UID: \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372420    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4qnz\" (UniqueName: \"kubernetes.io/projected/d1797edd-fa79-435c-9e29-3b93095377e1-kube-api-access-l4qnz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w\" (UID: \"d1797edd-fa79-435c-9e29-3b93095377e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372502    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1797edd-fa79-435c-9e29-3b93095377e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w\" (UID: \"d1797edd-fa79-435c-9e29-3b93095377e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372531    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8e28dd-5a24-420b-8233-38b38e2bfda3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4\" (UID: \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372587    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zh7\" (UniqueName: \"kubernetes.io/projected/db78128e-6cd0-445d-a667-075d7024ac7a-kube-api-access-h8zh7\") pod \"maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq\" (UID: \"db78128e-6cd0-445d-a667-075d7024ac7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372619    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9ct\" (UniqueName: \"kubernetes.io/projected/339358cd-ae97-49d2-a336-13be6d694007-kube-api-access-gj9ct\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7\" (UID: \"339358cd-ae97-49d2-a336-13be6d694007\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372659    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339358cd-ae97-49d2-a336-13be6d694007-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7\" (UID: \"339358cd-ae97-49d2-a336-13be6d694007\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:00.373249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.372785    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8pl\" (UniqueName: \"kubernetes.io/projected/f084d37b-3a9e-41b3-8820-ffa93587d8be-kube-api-access-px8pl\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw\" (UID: \"f084d37b-3a9e-41b3-8820-ffa93587d8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:00.376939 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.376491    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db78128e-6cd0-445d-a667-075d7024ac7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq\" (UID: \"db78128e-6cd0-445d-a667-075d7024ac7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:00.377796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.377766    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084d37b-3a9e-41b3-8820-ffa93587d8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw\" (UID: \"f084d37b-3a9e-41b3-8820-ffa93587d8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:00.378030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.377997    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c\" (UID: \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:00.378341 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.378318    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"]
Oct 30 23:00:00.378512 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.378423    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:00.385298 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.385236    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"]
Oct 30 23:00:00.434006 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.433914    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"]
Oct 30 23:00:00.450919 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.450702    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"]
Oct 30 23:00:00.470429 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.470290    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:00.471375 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.471350    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"]
Oct 30 23:00:00.478439 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.478409    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"]
Oct 30 23:00:00.480650 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480624    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8e28dd-5a24-420b-8233-38b38e2bfda3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4\" (UID: \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:00.480804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480673    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zh7\" (UniqueName: \"kubernetes.io/projected/db78128e-6cd0-445d-a667-075d7024ac7a-kube-api-access-h8zh7\") pod \"maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq\" (UID: \"db78128e-6cd0-445d-a667-075d7024ac7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:00.480804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480706    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26267bf8-2af8-45fc-8640-6a0a3812ee2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf\" (UID: \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:00.480804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480736    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3259e63-46a1-4572-9e46-b249d5fc24e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw\" (UID: \"e3259e63-46a1-4572-9e46-b249d5fc24e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:00.480804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480763    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9ct\" (UniqueName: \"kubernetes.io/projected/339358cd-ae97-49d2-a336-13be6d694007-kube-api-access-gj9ct\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7\" (UID: \"339358cd-ae97-49d2-a336-13be6d694007\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:00.480804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480802    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339358cd-ae97-49d2-a336-13be6d694007-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7\" (UID: \"339358cd-ae97-49d2-a336-13be6d694007\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:00.481106 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480857    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8pl\" (UniqueName: \"kubernetes.io/projected/f084d37b-3a9e-41b3-8820-ffa93587d8be-kube-api-access-px8pl\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw\" (UID: \"f084d37b-3a9e-41b3-8820-ffa93587d8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:00.481106 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480926    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/564d8a9f-e41c-41f5-8a03-90277087de3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs\" (UID: \"564d8a9f-e41c-41f5-8a03-90277087de3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:00.481106 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.480997    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmvv\" (UniqueName: \"kubernetes.io/projected/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-kube-api-access-rlmvv\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4\" (UID: \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:00.481106 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481083    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4\" (UID: \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:00.481402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481116    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db78128e-6cd0-445d-a667-075d7024ac7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq\" (UID: \"db78128e-6cd0-445d-a667-075d7024ac7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:00.481402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481163    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vx2\" (UniqueName: \"kubernetes.io/projected/564d8a9f-e41c-41f5-8a03-90277087de3f-kube-api-access-x9vx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs\" (UID: \"564d8a9f-e41c-41f5-8a03-90277087de3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:00.481402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481202    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwjt\" (UniqueName: \"kubernetes.io/projected/70232370-49f1-43ba-a855-c3ed07d277b8-kube-api-access-mcwjt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw\" (UID: \"70232370-49f1-43ba-a855-c3ed07d277b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:00.481402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481285    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70232370-49f1-43ba-a855-c3ed07d277b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw\" (UID: \"70232370-49f1-43ba-a855-c3ed07d277b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:00.481402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481315    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084d37b-3a9e-41b3-8820-ffa93587d8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw\" (UID: \"f084d37b-3a9e-41b3-8820-ffa93587d8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:00.481402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481367    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c\" (UID: \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:00.482440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481461    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nwh\" (UniqueName: \"kubernetes.io/projected/bb8e28dd-5a24-420b-8233-38b38e2bfda3-kube-api-access-m6nwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4\" (UID: \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:00.482440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481490    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8e28dd-5a24-420b-8233-38b38e2bfda3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4\" (UID: \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:00.482440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.481499    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4vw\" (UniqueName: \"kubernetes.io/projected/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-kube-api-access-4w4vw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c\" (UID: \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:00.482440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.482276    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db78128e-6cd0-445d-a667-075d7024ac7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq\" (UID: \"db78128e-6cd0-445d-a667-075d7024ac7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:00.482895 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.482874    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339358cd-ae97-49d2-a336-13be6d694007-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7\" (UID: \"339358cd-ae97-49d2-a336-13be6d694007\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:00.483036 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.482905    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084d37b-3a9e-41b3-8820-ffa93587d8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw\" (UID: \"f084d37b-3a9e-41b3-8820-ffa93587d8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:00.483120 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.482944    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4qnz\" (UniqueName: \"kubernetes.io/projected/d1797edd-fa79-435c-9e29-3b93095377e1-kube-api-access-l4qnz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w\" (UID: \"d1797edd-fa79-435c-9e29-3b93095377e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:00.483291 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.483265    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6bv\" (UniqueName: \"kubernetes.io/projected/e3259e63-46a1-4572-9e46-b249d5fc24e8-kube-api-access-mb6bv\") pod \"maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw\" (UID: \"e3259e63-46a1-4572-9e46-b249d5fc24e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:00.483291 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.483268    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c\" (UID: \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:00.483523 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.483337    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcfh\" (UniqueName: \"kubernetes.io/projected/26267bf8-2af8-45fc-8640-6a0a3812ee2f-kube-api-access-vfcfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf\" (UID: \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:00.483523 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.483379    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1797edd-fa79-435c-9e29-3b93095377e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w\" (UID: \"d1797edd-fa79-435c-9e29-3b93095377e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:00.484081 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.484061    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1797edd-fa79-435c-9e29-3b93095377e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w\" (UID: \"d1797edd-fa79-435c-9e29-3b93095377e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:00.491950 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.491930    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"]
Oct 30 23:00:00.492201 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.492179    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:00.501511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.501486    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"]
Oct 30 23:00:00.501804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.501779    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"]
Oct 30 23:00:00.502033 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.502019    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:00.503097 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.503076    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nwh\" (UniqueName: \"kubernetes.io/projected/bb8e28dd-5a24-420b-8233-38b38e2bfda3-kube-api-access-m6nwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4\" (UID: \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:00.503552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.503522    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4vw\" (UniqueName: \"kubernetes.io/projected/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-kube-api-access-4w4vw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c\" (UID: \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:00.503839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.503821    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zh7\" (UniqueName: \"kubernetes.io/projected/db78128e-6cd0-445d-a667-075d7024ac7a-kube-api-access-h8zh7\") pod \"maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq\" (UID: \"db78128e-6cd0-445d-a667-075d7024ac7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:00.505270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.504815    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8pl\" (UniqueName: \"kubernetes.io/projected/f084d37b-3a9e-41b3-8820-ffa93587d8be-kube-api-access-px8pl\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw\" (UID: \"f084d37b-3a9e-41b3-8820-ffa93587d8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:00.506062 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.506039    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9ct\" (UniqueName: \"kubernetes.io/projected/339358cd-ae97-49d2-a336-13be6d694007-kube-api-access-gj9ct\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7\" (UID: \"339358cd-ae97-49d2-a336-13be6d694007\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:00.506562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.506540    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"]
Oct 30 23:00:00.508937 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.508894    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4qnz\" (UniqueName: \"kubernetes.io/projected/d1797edd-fa79-435c-9e29-3b93095377e1-kube-api-access-l4qnz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w\" (UID: \"d1797edd-fa79-435c-9e29-3b93095377e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:00.512609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.512591    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:00.514341 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.514323    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"]
Oct 30 23:00:00.519727 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.519698    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"]
Oct 30 23:00:00.525638 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.525604    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"]
Oct 30 23:00:00.536478 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.536445    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"]
Oct 30 23:00:00.536769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.536572    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:00.542952 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.542928    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"]
Oct 30 23:00:00.543128 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.543059    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:00.548857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.548829    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"]
Oct 30 23:00:00.548998 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.548930    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:00.561865 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.561841    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"]
Oct 30 23:00:00.569787 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.569756    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"]
Oct 30 23:00:00.576183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.576154    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"]
Oct 30 23:00:00.582132 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.582101    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"]
Oct 30 23:00:00.584757 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.584726    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/564d8a9f-e41c-41f5-8a03-90277087de3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs\" (UID: \"564d8a9f-e41c-41f5-8a03-90277087de3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:00.584987 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.584967    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnq4q\" (UniqueName: \"kubernetes.io/projected/36d2e83a-4650-437c-a8fb-422c4931413f-kube-api-access-lnq4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h\" (UID: \"36d2e83a-4650-437c-a8fb-422c4931413f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:00.585152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.585124    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9p7j\" (UniqueName: \"kubernetes.io/projected/2a2d57df-edc4-4ffa-9308-4432b482085c-kube-api-access-m9p7j\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz\" (UID: \"2a2d57df-edc4-4ffa-9308-4432b482085c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:00.585337 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.585168    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmvv\" (UniqueName: \"kubernetes.io/projected/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-kube-api-access-rlmvv\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4\" (UID: \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:00.585337 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.585295    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4\" (UID: \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:00.585563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.585346    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vx2\" (UniqueName: \"kubernetes.io/projected/564d8a9f-e41c-41f5-8a03-90277087de3f-kube-api-access-x9vx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs\" (UID: \"564d8a9f-e41c-41f5-8a03-90277087de3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:00.585563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.585381    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwjt\" (UniqueName: \"kubernetes.io/projected/70232370-49f1-43ba-a855-c3ed07d277b8-kube-api-access-mcwjt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw\" (UID: \"70232370-49f1-43ba-a855-c3ed07d277b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:00.585563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.585421    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70232370-49f1-43ba-a855-c3ed07d277b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw\" (UID: \"70232370-49f1-43ba-a855-c3ed07d277b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:00.585563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.585454    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2671f5c-fcc0-46f8-8059-a8da58102f7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb\" (UID: \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:00.586070 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.585914    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ckl\" (UniqueName: \"kubernetes.io/projected/314f0b61-51e0-463d-8ee3-5e2cfc934690-kube-api-access-b6ckl\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn\" (UID: \"314f0b61-51e0-463d-8ee3-5e2cfc934690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:00.586070 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.586000    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/564d8a9f-e41c-41f5-8a03-90277087de3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs\" (UID: \"564d8a9f-e41c-41f5-8a03-90277087de3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:00.586245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.586192    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2d57df-edc4-4ffa-9308-4432b482085c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz\" (UID: \"2a2d57df-edc4-4ffa-9308-4432b482085c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:00.586342 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.586282    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmpz\" (UniqueName: \"kubernetes.io/projected/e2671f5c-fcc0-46f8-8059-a8da58102f7b-kube-api-access-hfmpz\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb\" (UID: \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:00.586342 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.586313    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zc7\" (UniqueName: \"kubernetes.io/projected/d7c80734-09bd-439c-baf5-95906d4a7bf0-kube-api-access-42zc7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd\" (UID: \"d7c80734-09bd-439c-baf5-95906d4a7bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:00.586455 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.586364    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d2e83a-4650-437c-a8fb-422c4931413f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h\" (UID: \"36d2e83a-4650-437c-a8fb-422c4931413f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:00.586642 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.586616    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70232370-49f1-43ba-a855-c3ed07d277b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw\" (UID: \"70232370-49f1-43ba-a855-c3ed07d277b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:00.588728 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.586766    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6bv\" (UniqueName: \"kubernetes.io/projected/e3259e63-46a1-4572-9e46-b249d5fc24e8-kube-api-access-mb6bv\") pod \"maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw\" (UID: \"e3259e63-46a1-4572-9e46-b249d5fc24e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:00.588971 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.588950    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcfh\" (UniqueName: \"kubernetes.io/projected/26267bf8-2af8-45fc-8640-6a0a3812ee2f-kube-api-access-vfcfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf\" (UID: \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:00.589509 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.589480    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314f0b61-51e0-463d-8ee3-5e2cfc934690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn\" (UID: \"314f0b61-51e0-463d-8ee3-5e2cfc934690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:00.589638 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.586898    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4\" (UID: \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:00.589740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.589684    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmll\" (UniqueName: \"kubernetes.io/projected/4663c26d-c795-4d52-9fdd-47b91bd76fe1-kube-api-access-rlmll\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb\" (UID: \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:00.589922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.589904    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26267bf8-2af8-45fc-8640-6a0a3812ee2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf\" (UID: \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:00.590018 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.589759    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"]
Oct 30 23:00:00.590191 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.590025    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3259e63-46a1-4572-9e46-b249d5fc24e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw\" (UID: \"e3259e63-46a1-4572-9e46-b249d5fc24e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:00.590410 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.590393    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c80734-09bd-439c-baf5-95906d4a7bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd\" (UID: \"d7c80734-09bd-439c-baf5-95906d4a7bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:00.590542 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.590528    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4663c26d-c795-4d52-9fdd-47b91bd76fe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb\" (UID: \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:00.591028 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.590998    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3259e63-46a1-4572-9e46-b249d5fc24e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw\" (UID: \"e3259e63-46a1-4572-9e46-b249d5fc24e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:00.591623 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.591590    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26267bf8-2af8-45fc-8640-6a0a3812ee2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf\" (UID: \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:00.598870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.598842    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"]
Oct 30 23:00:00.599449 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.599431    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:00.607042 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.606999    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"]
Oct 30 23:00:00.612287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.607915    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwjt\" (UniqueName: \"kubernetes.io/projected/70232370-49f1-43ba-a855-c3ed07d277b8-kube-api-access-mcwjt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw\" (UID: \"70232370-49f1-43ba-a855-c3ed07d277b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:00.612287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.609873    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcfh\" (UniqueName: \"kubernetes.io/projected/26267bf8-2af8-45fc-8640-6a0a3812ee2f-kube-api-access-vfcfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf\" (UID: \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:00.612287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.611841    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vx2\" (UniqueName: \"kubernetes.io/projected/564d8a9f-e41c-41f5-8a03-90277087de3f-kube-api-access-x9vx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs\" (UID: \"564d8a9f-e41c-41f5-8a03-90277087de3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:00.614935 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.614905    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmvv\" (UniqueName: \"kubernetes.io/projected/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-kube-api-access-rlmvv\") pod \"maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4\" (UID: \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:00.615065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.614974    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6bv\" (UniqueName: \"kubernetes.io/projected/e3259e63-46a1-4572-9e46-b249d5fc24e8-kube-api-access-mb6bv\") pod \"maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw\" (UID: \"e3259e63-46a1-4572-9e46-b249d5fc24e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:00.623691 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.623662    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"]
Oct 30 23:00:00.634183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.634154    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:00.677173 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.677049    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692422    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9p7j\" (UniqueName: \"kubernetes.io/projected/2a2d57df-edc4-4ffa-9308-4432b482085c-kube-api-access-m9p7j\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz\" (UID: \"2a2d57df-edc4-4ffa-9308-4432b482085c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692592    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2671f5c-fcc0-46f8-8059-a8da58102f7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb\" (UID: \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692647    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ckl\" (UniqueName: \"kubernetes.io/projected/314f0b61-51e0-463d-8ee3-5e2cfc934690-kube-api-access-b6ckl\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn\" (UID: \"314f0b61-51e0-463d-8ee3-5e2cfc934690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692689    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2d57df-edc4-4ffa-9308-4432b482085c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz\" (UID: \"2a2d57df-edc4-4ffa-9308-4432b482085c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692741    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmpz\" (UniqueName: \"kubernetes.io/projected/e2671f5c-fcc0-46f8-8059-a8da58102f7b-kube-api-access-hfmpz\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb\" (UID: \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692766    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zc7\" (UniqueName: \"kubernetes.io/projected/d7c80734-09bd-439c-baf5-95906d4a7bf0-kube-api-access-42zc7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd\" (UID: \"d7c80734-09bd-439c-baf5-95906d4a7bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692796    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d2e83a-4650-437c-a8fb-422c4931413f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h\" (UID: \"36d2e83a-4650-437c-a8fb-422c4931413f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692889    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314f0b61-51e0-463d-8ee3-5e2cfc934690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn\" (UID: \"314f0b61-51e0-463d-8ee3-5e2cfc934690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692921    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmll\" (UniqueName: \"kubernetes.io/projected/4663c26d-c795-4d52-9fdd-47b91bd76fe1-kube-api-access-rlmll\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb\" (UID: \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.692996    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c80734-09bd-439c-baf5-95906d4a7bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd\" (UID: \"d7c80734-09bd-439c-baf5-95906d4a7bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.693024    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4663c26d-c795-4d52-9fdd-47b91bd76fe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb\" (UID: \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.693109    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnq4q\" (UniqueName: \"kubernetes.io/projected/36d2e83a-4650-437c-a8fb-422c4931413f-kube-api-access-lnq4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h\" (UID: \"36d2e83a-4650-437c-a8fb-422c4931413f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:00.694847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.694499    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2671f5c-fcc0-46f8-8059-a8da58102f7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb\" (UID: \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:00.700256 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.697524    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314f0b61-51e0-463d-8ee3-5e2cfc934690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn\" (UID: \"314f0b61-51e0-463d-8ee3-5e2cfc934690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:00.700256 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.698743    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d2e83a-4650-437c-a8fb-422c4931413f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h\" (UID: \"36d2e83a-4650-437c-a8fb-422c4931413f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:00.700256 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.699460    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c80734-09bd-439c-baf5-95906d4a7bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd\" (UID: \"d7c80734-09bd-439c-baf5-95906d4a7bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:00.700256 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.700069    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4663c26d-c795-4d52-9fdd-47b91bd76fe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb\" (UID: \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:00.709274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.703955    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2d57df-edc4-4ffa-9308-4432b482085c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz\" (UID: \"2a2d57df-edc4-4ffa-9308-4432b482085c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:00.716377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.715899    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zc7\" (UniqueName: \"kubernetes.io/projected/d7c80734-09bd-439c-baf5-95906d4a7bf0-kube-api-access-42zc7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd\" (UID: \"d7c80734-09bd-439c-baf5-95906d4a7bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:00.730467 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.728085    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:00.735244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.735141    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ckl\" (UniqueName: \"kubernetes.io/projected/314f0b61-51e0-463d-8ee3-5e2cfc934690-kube-api-access-b6ckl\") pod \"maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn\" (UID: \"314f0b61-51e0-463d-8ee3-5e2cfc934690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:00.748057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.748023    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnq4q\" (UniqueName: \"kubernetes.io/projected/36d2e83a-4650-437c-a8fb-422c4931413f-kube-api-access-lnq4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h\" (UID: \"36d2e83a-4650-437c-a8fb-422c4931413f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:00.748691 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.748666    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:00.750652 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.750618    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9p7j\" (UniqueName: \"kubernetes.io/projected/2a2d57df-edc4-4ffa-9308-4432b482085c-kube-api-access-m9p7j\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz\" (UID: \"2a2d57df-edc4-4ffa-9308-4432b482085c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:00.764242 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.762687    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:00.770703 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.770677    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmpz\" (UniqueName: \"kubernetes.io/projected/e2671f5c-fcc0-46f8-8059-a8da58102f7b-kube-api-access-hfmpz\") pod \"maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb\" (UID: \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:00.771739 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.771608    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:00.776491 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.776461    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmll\" (UniqueName: \"kubernetes.io/projected/4663c26d-c795-4d52-9fdd-47b91bd76fe1-kube-api-access-rlmll\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb\" (UID: \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:00.785911 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.785884    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:00.799469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.799435    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:00.808140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.808103    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:00.859968 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.859392    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:00.881473 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.879121    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:00.888146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.888118    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:00.900991 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.900953    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:00.916358 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:00.916256    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:01.005483 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.004297    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:01.057548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.053497    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:01.176957 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.176907    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"]
Oct 30 23:00:01.288101 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:01.288020    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb78128e_6cd0_445d_a667_075d7024ac7a.slice/crio-a37b5c2619ffeba1d74ecd5cc29d954dda19504fff8b2d1f28b6320cb453325f WatchSource:0}: Error finding container a37b5c2619ffeba1d74ecd5cc29d954dda19504fff8b2d1f28b6320cb453325f: Status 404 returned error can't find the container with id a37b5c2619ffeba1d74ecd5cc29d954dda19504fff8b2d1f28b6320cb453325f
Oct 30 23:00:01.592245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.592196    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"]
Oct 30 23:00:01.594923 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:01.594750    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e6e4c4_0302_41c1_a6b9_a71a30e7e2b9.slice/crio-bbe8bfa3e3ec521deda2837d922570b39646a54af1bd2d1e42c8430441f2dd11 WatchSource:0}: Error finding container bbe8bfa3e3ec521deda2837d922570b39646a54af1bd2d1e42c8430441f2dd11: Status 404 returned error can't find the container with id bbe8bfa3e3ec521deda2837d922570b39646a54af1bd2d1e42c8430441f2dd11
Oct 30 23:00:01.637514 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.637468    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"]
Oct 30 23:00:01.641985 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:01.641952    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3259e63_46a1_4572_9e46_b249d5fc24e8.slice/crio-90b3775e610d8eb468e3aa84b8d1a77fcc5b3188e7b4d984c994c75a0fb69ae7 WatchSource:0}: Error finding container 90b3775e610d8eb468e3aa84b8d1a77fcc5b3188e7b4d984c994c75a0fb69ae7: Status 404 returned error can't find the container with id 90b3775e610d8eb468e3aa84b8d1a77fcc5b3188e7b4d984c994c75a0fb69ae7
Oct 30 23:00:01.723879 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.723850    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"]
Oct 30 23:00:01.731637 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.731604    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"]
Oct 30 23:00:01.740101 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:01.740070    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8e28dd_5a24_420b_8233_38b38e2bfda3.slice/crio-0cfd5418bf541005469749bec479c29a7d6fa9e8f5426bf23a1591b4992c5563 WatchSource:0}: Error finding container 0cfd5418bf541005469749bec479c29a7d6fa9e8f5426bf23a1591b4992c5563: Status 404 returned error can't find the container with id 0cfd5418bf541005469749bec479c29a7d6fa9e8f5426bf23a1591b4992c5563
Oct 30 23:00:01.740504 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:01.740485    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339358cd_ae97_49d2_a336_13be6d694007.slice/crio-e7adb13b8badae8639d24a2dafc366b7ac4fbe8ec195b8c568275257c2a58a63 WatchSource:0}: Error finding container e7adb13b8badae8639d24a2dafc366b7ac4fbe8ec195b8c568275257c2a58a63: Status 404 returned error can't find the container with id e7adb13b8badae8639d24a2dafc366b7ac4fbe8ec195b8c568275257c2a58a63
Oct 30 23:00:01.765026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.764990    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"]
Oct 30 23:00:01.777888 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.777856    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"]
Oct 30 23:00:01.784892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.784857    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"]
Oct 30 23:00:01.791310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.791289    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"]
Oct 30 23:00:01.828561 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.828530    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"]
Oct 30 23:00:01.870039 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:01.869989    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1797edd_fa79_435c_9e29_3b93095377e1.slice/crio-8067b430ce826088e93d1ae6201a231da40528813c8baf6a54068071b5169c51 WatchSource:0}: Error finding container 8067b430ce826088e93d1ae6201a231da40528813c8baf6a54068071b5169c51: Status 404 returned error can't find the container with id 8067b430ce826088e93d1ae6201a231da40528813c8baf6a54068071b5169c51
Oct 30 23:00:01.874870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.874844    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"]
Oct 30 23:00:01.877794 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:01.877770    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26267bf8_2af8_45fc_8640_6a0a3812ee2f.slice/crio-d2a9ab6f24f5241ce585be40840284748b48e35df84a908c7e4a139312f5755e WatchSource:0}: Error finding container d2a9ab6f24f5241ce585be40840284748b48e35df84a908c7e4a139312f5755e: Status 404 returned error can't find the container with id d2a9ab6f24f5241ce585be40840284748b48e35df84a908c7e4a139312f5755e
Oct 30 23:00:01.899481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.899455    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"]
Oct 30 23:00:01.910497 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.910473    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"]
Oct 30 23:00:01.996963 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:01.996696    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"]
Oct 30 23:00:02.006414 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.006385    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"]
Oct 30 23:00:02.039046 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.039009    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf" event={"ID":"26267bf8-2af8-45fc-8640-6a0a3812ee2f","Type":"ContainerStarted","Data":"d2a9ab6f24f5241ce585be40840284748b48e35df84a908c7e4a139312f5755e"}
Oct 30 23:00:02.040491 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.040460    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" event={"ID":"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9","Type":"ContainerStarted","Data":"bbe8bfa3e3ec521deda2837d922570b39646a54af1bd2d1e42c8430441f2dd11"}
Oct 30 23:00:02.041925 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.041897    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" event={"ID":"339358cd-ae97-49d2-a336-13be6d694007","Type":"ContainerStarted","Data":"e7adb13b8badae8639d24a2dafc366b7ac4fbe8ec195b8c568275257c2a58a63"}
Oct 30 23:00:02.044168 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.044144    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" event={"ID":"db78128e-6cd0-445d-a667-075d7024ac7a","Type":"ContainerStarted","Data":"bf0f68e972c178bc97fd0fb9438c9613989fe457841fb247fed4c465f7c422d2"}
Oct 30 23:00:02.044328 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.044191    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" event={"ID":"db78128e-6cd0-445d-a667-075d7024ac7a","Type":"ContainerStarted","Data":"a37b5c2619ffeba1d74ecd5cc29d954dda19504fff8b2d1f28b6320cb453325f"}
Oct 30 23:00:02.045676 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.045654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" event={"ID":"4663c26d-c795-4d52-9fdd-47b91bd76fe1","Type":"ContainerStarted","Data":"0f85897c95a50fd85676edb389b7f65b33268b9cf35b6c8d7e6716f223bc6db2"}
Oct 30 23:00:02.047596 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.047574    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" event={"ID":"564d8a9f-e41c-41f5-8a03-90277087de3f","Type":"ContainerStarted","Data":"77d07478ef6f8a7a082aad6d1b188996084592899611fbefaa1960d29d6690c3"}
Oct 30 23:00:02.048298 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.048277    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"]
Oct 30 23:00:02.048963 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.048938    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" event={"ID":"70232370-49f1-43ba-a855-c3ed07d277b8","Type":"ContainerStarted","Data":"3a76e06ceb260b1117b7cbdbccf82dc5aceec007c36bdcf53b6f71bbe8f58c8f"}
Oct 30 23:00:02.050313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.050291    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4" event={"ID":"aaa9064d-b09a-4b48-b007-0cf8e3c1efac","Type":"ContainerStarted","Data":"e54ee8f55168a245343c3704efb85b7ec0654f9f16608f16d318075a9adeb9eb"}
Oct 30 23:00:02.054663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.054636    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"]
Oct 30 23:00:02.055085 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.055054    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" event={"ID":"e2671f5c-fcc0-46f8-8059-a8da58102f7b","Type":"ContainerStarted","Data":"9e1c386589f5d932a48dc6e13b99785faf7294a5b9254ebe9014927e7ff46103"}
Oct 30 23:00:02.056503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.056481    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" event={"ID":"f084d37b-3a9e-41b3-8820-ffa93587d8be","Type":"ContainerStarted","Data":"46ea04aeda8ec49654ebe4a7b5686d083adbfbe43b6bbf3e58dcd49e1b3f038c"}
Oct 30 23:00:02.058055 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.058031    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" event={"ID":"bb8e28dd-5a24-420b-8233-38b38e2bfda3","Type":"ContainerStarted","Data":"0cfd5418bf541005469749bec479c29a7d6fa9e8f5426bf23a1591b4992c5563"}
Oct 30 23:00:02.059409 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.059387    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" event={"ID":"e3259e63-46a1-4572-9e46-b249d5fc24e8","Type":"ContainerStarted","Data":"90b3775e610d8eb468e3aa84b8d1a77fcc5b3188e7b4d984c994c75a0fb69ae7"}
Oct 30 23:00:02.060844 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.060818    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" event={"ID":"d1797edd-fa79-435c-9e29-3b93095377e1","Type":"ContainerStarted","Data":"8067b430ce826088e93d1ae6201a231da40528813c8baf6a54068071b5169c51"}
Oct 30 23:00:02.066323 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:02.066296    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314f0b61_51e0_463d_8ee3_5e2cfc934690.slice/crio-3e0dcb575acb1e6567c365464a3bef1c2d60f6f74022aeb44be4698a7e8fbcde WatchSource:0}: Error finding container 3e0dcb575acb1e6567c365464a3bef1c2d60f6f74022aeb44be4698a7e8fbcde: Status 404 returned error can't find the container with id 3e0dcb575acb1e6567c365464a3bef1c2d60f6f74022aeb44be4698a7e8fbcde
Oct 30 23:00:02.088803 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:02.088726    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" podStartSLOduration=2.088707722 podStartE2EDuration="2.088707722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:02.076546873 +0000 UTC m=+3186.115485769" watchObservedRunningTime="2026-10-30 23:00:02.088707722 +0000 UTC m=+3186.127646609"
Oct 30 23:00:03.081024 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.080987    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" event={"ID":"e2671f5c-fcc0-46f8-8059-a8da58102f7b","Type":"ContainerStarted","Data":"e301a63944b6d8b2f9aa9b3c98f756dfe7c1b8b296b4aec784acbc869b5b19ef"}
Oct 30 23:00:03.084914 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.084619    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" event={"ID":"f084d37b-3a9e-41b3-8820-ffa93587d8be","Type":"ContainerStarted","Data":"e86dd0cbf64b73a4e67a5bd87805f08d33e641e728fd05262aa63d734b5ac36b"}
Oct 30 23:00:03.087842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.087506    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" event={"ID":"bb8e28dd-5a24-420b-8233-38b38e2bfda3","Type":"ContainerStarted","Data":"629d845db89c7e386ad943602d9c8946df266ce78478a82a80a9fda2adcacaf5"}
Oct 30 23:00:03.091267 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.091205    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" event={"ID":"e3259e63-46a1-4572-9e46-b249d5fc24e8","Type":"ContainerStarted","Data":"30133137576a664fd29fb26d03cde3dd08acbbae8d82d4862dc11af0c59d2ca1"}
Oct 30 23:00:03.091571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.091549    4581 scope.go:122] "RemoveContainer" containerID="30133137576a664fd29fb26d03cde3dd08acbbae8d82d4862dc11af0c59d2ca1"
Oct 30 23:00:03.093527 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.093475    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" event={"ID":"d1797edd-fa79-435c-9e29-3b93095377e1","Type":"ContainerStarted","Data":"c219713df36f302fd761b810def99897cfb3aead91304ae73d57e4f709caf7ab"}
Oct 30 23:00:03.096469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.096390    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" event={"ID":"36d2e83a-4650-437c-a8fb-422c4931413f","Type":"ContainerStarted","Data":"e4a9f10e0ae65913c9f471a956af3df504442b44d0493f6fd5b4c7f3025117d5"}
Oct 30 23:00:03.098703 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.098285    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" event={"ID":"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9","Type":"ContainerStarted","Data":"2174b8faf9196800c9ab9e036bfee4f7e04f44740d3bf5df4da9b69418ec3d07"}
Oct 30 23:00:03.100669 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.100639    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" event={"ID":"339358cd-ae97-49d2-a336-13be6d694007","Type":"ContainerStarted","Data":"0eacf17682d3ed649a4ebb94af20464fc1d4f7a405143fb8c429c637a22d5c41"}
Oct 30 23:00:03.102771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.102750    4581 generic.go:356] "Generic (PLEG): container finished" podID="db78128e-6cd0-445d-a667-075d7024ac7a" containerID="bf0f68e972c178bc97fd0fb9438c9613989fe457841fb247fed4c465f7c422d2" exitCode=60
Oct 30 23:00:03.102862 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.102850    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" event={"ID":"db78128e-6cd0-445d-a667-075d7024ac7a","Type":"ContainerDied","Data":"bf0f68e972c178bc97fd0fb9438c9613989fe457841fb247fed4c465f7c422d2"}
Oct 30 23:00:03.103103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.103089    4581 scope.go:122] "RemoveContainer" containerID="bf0f68e972c178bc97fd0fb9438c9613989fe457841fb247fed4c465f7c422d2"
Oct 30 23:00:03.105004 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.104986    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" event={"ID":"4663c26d-c795-4d52-9fdd-47b91bd76fe1","Type":"ContainerStarted","Data":"a782739235a6ec58cdb93987479f1142902ab4fb59ea4c98e97de99adfb242d2"}
Oct 30 23:00:03.107006 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.106985    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" event={"ID":"564d8a9f-e41c-41f5-8a03-90277087de3f","Type":"ContainerStarted","Data":"cafe5ccd2d7e40f82cfba27fb1fe5060d5b99e2912dff461a6e5476769c9c1b3"}
Oct 30 23:00:03.109101 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.109080    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" event={"ID":"314f0b61-51e0-463d-8ee3-5e2cfc934690","Type":"ContainerStarted","Data":"3e0dcb575acb1e6567c365464a3bef1c2d60f6f74022aeb44be4698a7e8fbcde"}
Oct 30 23:00:03.112886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.112859    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" event={"ID":"d7c80734-09bd-439c-baf5-95906d4a7bf0","Type":"ContainerStarted","Data":"c1fb0eac8379045618bb96d6b6fd1009f595a164d45f9c1550d845a4743195f2"}
Oct 30 23:00:03.116399 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.115956    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" podStartSLOduration=3.115940919 podStartE2EDuration="3.115940919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.114040188 +0000 UTC m=+3187.152979074" watchObservedRunningTime="2026-10-30 23:00:03.115940919 +0000 UTC m=+3187.154879805"
Oct 30 23:00:03.116549 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.116407    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" event={"ID":"70232370-49f1-43ba-a855-c3ed07d277b8","Type":"ContainerStarted","Data":"a6b70d4abad04c05df0bd7f55d2cd1471b0617b435782db790085247e45c3459"}
Oct 30 23:00:03.128910 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.128876    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" event={"ID":"2a2d57df-edc4-4ffa-9308-4432b482085c","Type":"ContainerStarted","Data":"57bc91c26f6f4d2d1dd9c4e9f58dad3e43e18cbc14f1285af565421dd2f4edde"}
Oct 30 23:00:03.203548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.203456    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" podStartSLOduration=3.203437247 podStartE2EDuration="3.203437247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.196415713 +0000 UTC m=+3187.235354598" watchObservedRunningTime="2026-10-30 23:00:03.203437247 +0000 UTC m=+3187.242376136"
Oct 30 23:00:03.283034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.282975    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" podStartSLOduration=3.282954923 podStartE2EDuration="3.282954923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.264820556 +0000 UTC m=+3187.303759442" watchObservedRunningTime="2026-10-30 23:00:03.282954923 +0000 UTC m=+3187.321893810"
Oct 30 23:00:03.302124 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.302065    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" podStartSLOduration=3.3020478349999998 podStartE2EDuration="3.302047835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.301457977 +0000 UTC m=+3187.340396854" watchObservedRunningTime="2026-10-30 23:00:03.302047835 +0000 UTC m=+3187.340986722"
Oct 30 23:00:03.360235 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.359686    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" podStartSLOduration=3.359661417 podStartE2EDuration="3.359661417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.344452116 +0000 UTC m=+3187.383391001" watchObservedRunningTime="2026-10-30 23:00:03.359661417 +0000 UTC m=+3187.398600304"
Oct 30 23:00:03.399352 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.399211    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" podStartSLOduration=3.399199018 podStartE2EDuration="3.399199018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.395709359 +0000 UTC m=+3187.434648245" watchObservedRunningTime="2026-10-30 23:00:03.399199018 +0000 UTC m=+3187.438137903"
Oct 30 23:00:03.442955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.442900    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" podStartSLOduration=3.442882712 podStartE2EDuration="3.442882712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.437352605 +0000 UTC m=+3187.476291492" watchObservedRunningTime="2026-10-30 23:00:03.442882712 +0000 UTC m=+3187.481821599"
Oct 30 23:00:03.480264 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.480178    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" podStartSLOduration=3.480160339 podStartE2EDuration="3.480160339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.471958146 +0000 UTC m=+3187.510897033" watchObservedRunningTime="2026-10-30 23:00:03.480160339 +0000 UTC m=+3187.519099224"
Oct 30 23:00:03.508481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.508411    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" podStartSLOduration=3.508396084 podStartE2EDuration="3.508396084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.504678211 +0000 UTC m=+3187.543617093" watchObservedRunningTime="2026-10-30 23:00:03.508396084 +0000 UTC m=+3187.547334970"
Oct 30 23:00:03.542962 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.542907    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" podStartSLOduration=3.542893063 podStartE2EDuration="3.542893063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.537404859 +0000 UTC m=+3187.576343745" watchObservedRunningTime="2026-10-30 23:00:03.542893063 +0000 UTC m=+3187.581831949"
Oct 30 23:00:03.573931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:03.573613    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" podStartSLOduration=3.573593994 podStartE2EDuration="3.573593994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:03.571341188 +0000 UTC m=+3187.610280075" watchObservedRunningTime="2026-10-30 23:00:03.573593994 +0000 UTC m=+3187.612532882"
Oct 30 23:00:04.146057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.146020    4581 generic.go:356] "Generic (PLEG): container finished" podID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerID="a782739235a6ec58cdb93987479f1142902ab4fb59ea4c98e97de99adfb242d2" exitCode=60
Oct 30 23:00:04.146537 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.146110    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" event={"ID":"4663c26d-c795-4d52-9fdd-47b91bd76fe1","Type":"ContainerDied","Data":"a782739235a6ec58cdb93987479f1142902ab4fb59ea4c98e97de99adfb242d2"}
Oct 30 23:00:04.146537 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.146497    4581 scope.go:122] "RemoveContainer" containerID="a782739235a6ec58cdb93987479f1142902ab4fb59ea4c98e97de99adfb242d2"
Oct 30 23:00:04.148321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.148299    4581 generic.go:356] "Generic (PLEG): container finished" podID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerID="cafe5ccd2d7e40f82cfba27fb1fe5060d5b99e2912dff461a6e5476769c9c1b3" exitCode=60
Oct 30 23:00:04.148434 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.148377    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" event={"ID":"564d8a9f-e41c-41f5-8a03-90277087de3f","Type":"ContainerDied","Data":"cafe5ccd2d7e40f82cfba27fb1fe5060d5b99e2912dff461a6e5476769c9c1b3"}
Oct 30 23:00:04.148648 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.148628    4581 scope.go:122] "RemoveContainer" containerID="cafe5ccd2d7e40f82cfba27fb1fe5060d5b99e2912dff461a6e5476769c9c1b3"
Oct 30 23:00:04.150813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.150791    4581 generic.go:356] "Generic (PLEG): container finished" podID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerID="90d411ce2ae17ff0a8791cc62c54b6aed65ce93d1317b3cd274bd818fcd9716d" exitCode=60
Oct 30 23:00:04.151037 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.151014    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" event={"ID":"314f0b61-51e0-463d-8ee3-5e2cfc934690","Type":"ContainerDied","Data":"90d411ce2ae17ff0a8791cc62c54b6aed65ce93d1317b3cd274bd818fcd9716d"}
Oct 30 23:00:04.151364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.151145    4581 scope.go:122] "RemoveContainer" containerID="90d411ce2ae17ff0a8791cc62c54b6aed65ce93d1317b3cd274bd818fcd9716d"
Oct 30 23:00:04.155830 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.155803    4581 generic.go:356] "Generic (PLEG): container finished" podID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerID="b8e66c9c3b81bf53605a9fc0d217ac0674025f0a364cd26da57553b194c9f9b7" exitCode=60
Oct 30 23:00:04.155947 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.155898    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" event={"ID":"d7c80734-09bd-439c-baf5-95906d4a7bf0","Type":"ContainerDied","Data":"b8e66c9c3b81bf53605a9fc0d217ac0674025f0a364cd26da57553b194c9f9b7"}
Oct 30 23:00:04.156165 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.156139    4581 scope.go:122] "RemoveContainer" containerID="b8e66c9c3b81bf53605a9fc0d217ac0674025f0a364cd26da57553b194c9f9b7"
Oct 30 23:00:04.158906 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.158877    4581 generic.go:356] "Generic (PLEG): container finished" podID="70232370-49f1-43ba-a855-c3ed07d277b8" containerID="a6b70d4abad04c05df0bd7f55d2cd1471b0617b435782db790085247e45c3459" exitCode=60
Oct 30 23:00:04.159067 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.159035    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" event={"ID":"70232370-49f1-43ba-a855-c3ed07d277b8","Type":"ContainerDied","Data":"a6b70d4abad04c05df0bd7f55d2cd1471b0617b435782db790085247e45c3459"}
Oct 30 23:00:04.159360 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.159341    4581 scope.go:122] "RemoveContainer" containerID="a6b70d4abad04c05df0bd7f55d2cd1471b0617b435782db790085247e45c3459"
Oct 30 23:00:04.161385 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.161361    4581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerID="e7d65f89dd7c823b79d2cc7258820940a65f9248d858259610a1ed1cb46c6937" exitCode=60
Oct 30 23:00:04.161490 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.161465    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4" event={"ID":"aaa9064d-b09a-4b48-b007-0cf8e3c1efac","Type":"ContainerDied","Data":"e7d65f89dd7c823b79d2cc7258820940a65f9248d858259610a1ed1cb46c6937"}
Oct 30 23:00:04.161904 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.161814    4581 scope.go:122] "RemoveContainer" containerID="e7d65f89dd7c823b79d2cc7258820940a65f9248d858259610a1ed1cb46c6937"
Oct 30 23:00:04.167721 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.167696    4581 generic.go:356] "Generic (PLEG): container finished" podID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerID="d2690122e7e98fa8e45da0eca22066103325c47e4a15b3ac3faa6d542cb20cb1" exitCode=60
Oct 30 23:00:04.167865 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.167847    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" event={"ID":"2a2d57df-edc4-4ffa-9308-4432b482085c","Type":"ContainerDied","Data":"d2690122e7e98fa8e45da0eca22066103325c47e4a15b3ac3faa6d542cb20cb1"}
Oct 30 23:00:04.168088 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.168067    4581 scope.go:122] "RemoveContainer" containerID="d2690122e7e98fa8e45da0eca22066103325c47e4a15b3ac3faa6d542cb20cb1"
Oct 30 23:00:04.170308 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.170285    4581 generic.go:356] "Generic (PLEG): container finished" podID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerID="e301a63944b6d8b2f9aa9b3c98f756dfe7c1b8b296b4aec784acbc869b5b19ef" exitCode=60
Oct 30 23:00:04.170436 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.170360    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" event={"ID":"e2671f5c-fcc0-46f8-8059-a8da58102f7b","Type":"ContainerDied","Data":"e301a63944b6d8b2f9aa9b3c98f756dfe7c1b8b296b4aec784acbc869b5b19ef"}
Oct 30 23:00:04.170661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.170642    4581 scope.go:122] "RemoveContainer" containerID="e301a63944b6d8b2f9aa9b3c98f756dfe7c1b8b296b4aec784acbc869b5b19ef"
Oct 30 23:00:04.172356 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.172331    4581 generic.go:356] "Generic (PLEG): container finished" podID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerID="e86dd0cbf64b73a4e67a5bd87805f08d33e641e728fd05262aa63d734b5ac36b" exitCode=60
Oct 30 23:00:04.172459 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.172413    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" event={"ID":"f084d37b-3a9e-41b3-8820-ffa93587d8be","Type":"ContainerDied","Data":"e86dd0cbf64b73a4e67a5bd87805f08d33e641e728fd05262aa63d734b5ac36b"}
Oct 30 23:00:04.172643 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.172627    4581 scope.go:122] "RemoveContainer" containerID="e86dd0cbf64b73a4e67a5bd87805f08d33e641e728fd05262aa63d734b5ac36b"
Oct 30 23:00:04.175568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.175515    4581 generic.go:356] "Generic (PLEG): container finished" podID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerID="629d845db89c7e386ad943602d9c8946df266ce78478a82a80a9fda2adcacaf5" exitCode=60
Oct 30 23:00:04.175662 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.175594    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" event={"ID":"bb8e28dd-5a24-420b-8233-38b38e2bfda3","Type":"ContainerDied","Data":"629d845db89c7e386ad943602d9c8946df266ce78478a82a80a9fda2adcacaf5"}
Oct 30 23:00:04.175843 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.175826    4581 scope.go:122] "RemoveContainer" containerID="629d845db89c7e386ad943602d9c8946df266ce78478a82a80a9fda2adcacaf5"
Oct 30 23:00:04.200990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.200692    4581 generic.go:356] "Generic (PLEG): container finished" podID="e3259e63-46a1-4572-9e46-b249d5fc24e8" containerID="30133137576a664fd29fb26d03cde3dd08acbbae8d82d4862dc11af0c59d2ca1" exitCode=60
Oct 30 23:00:04.200990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.200806    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" event={"ID":"e3259e63-46a1-4572-9e46-b249d5fc24e8","Type":"ContainerDied","Data":"30133137576a664fd29fb26d03cde3dd08acbbae8d82d4862dc11af0c59d2ca1"}
Oct 30 23:00:04.200990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.200854    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" event={"ID":"e3259e63-46a1-4572-9e46-b249d5fc24e8","Type":"ContainerStarted","Data":"584b797514918227bc688aefd445e6b4b0cfaab14d5f4275007b6cf1081d7c87"}
Oct 30 23:00:04.200990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.200887    4581 scope.go:122] "RemoveContainer" containerID="30133137576a664fd29fb26d03cde3dd08acbbae8d82d4862dc11af0c59d2ca1"
Oct 30 23:00:04.201252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.201183    4581 scope.go:122] "RemoveContainer" containerID="584b797514918227bc688aefd445e6b4b0cfaab14d5f4275007b6cf1081d7c87"
Oct 30 23:00:04.201627 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:04.201591    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw_odh-ai-gateway-infra(e3259e63-46a1-4572-9e46-b249d5fc24e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8"
Oct 30 23:00:04.207294 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.207267    4581 generic.go:356] "Generic (PLEG): container finished" podID="d1797edd-fa79-435c-9e29-3b93095377e1" containerID="c219713df36f302fd761b810def99897cfb3aead91304ae73d57e4f709caf7ab" exitCode=60
Oct 30 23:00:04.207495 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.207472    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" event={"ID":"d1797edd-fa79-435c-9e29-3b93095377e1","Type":"ContainerDied","Data":"c219713df36f302fd761b810def99897cfb3aead91304ae73d57e4f709caf7ab"}
Oct 30 23:00:04.207970 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.207946    4581 scope.go:122] "RemoveContainer" containerID="c219713df36f302fd761b810def99897cfb3aead91304ae73d57e4f709caf7ab"
Oct 30 23:00:04.210654 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.210619    4581 generic.go:356] "Generic (PLEG): container finished" podID="36d2e83a-4650-437c-a8fb-422c4931413f" containerID="4b0cd5bd043db0707d2fa75a506639bda66ef811d01f4dbba389fc286a2545e6" exitCode=60
Oct 30 23:00:04.210753 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.210709    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" event={"ID":"36d2e83a-4650-437c-a8fb-422c4931413f","Type":"ContainerDied","Data":"4b0cd5bd043db0707d2fa75a506639bda66ef811d01f4dbba389fc286a2545e6"}
Oct 30 23:00:04.210992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.210967    4581 scope.go:122] "RemoveContainer" containerID="4b0cd5bd043db0707d2fa75a506639bda66ef811d01f4dbba389fc286a2545e6"
Oct 30 23:00:04.212801 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.212781    4581 generic.go:356] "Generic (PLEG): container finished" podID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerID="e1a216a90b08c24d06bb01aa2980d0103ce9054eece706343f974540dbb66268" exitCode=60
Oct 30 23:00:04.212908 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.212801    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf" event={"ID":"26267bf8-2af8-45fc-8640-6a0a3812ee2f","Type":"ContainerDied","Data":"e1a216a90b08c24d06bb01aa2980d0103ce9054eece706343f974540dbb66268"}
Oct 30 23:00:04.213087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.213071    4581 scope.go:122] "RemoveContainer" containerID="e1a216a90b08c24d06bb01aa2980d0103ce9054eece706343f974540dbb66268"
Oct 30 23:00:04.215112 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.215093    4581 generic.go:356] "Generic (PLEG): container finished" podID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerID="2174b8faf9196800c9ab9e036bfee4f7e04f44740d3bf5df4da9b69418ec3d07" exitCode=60
Oct 30 23:00:04.215331 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.215311    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" event={"ID":"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9","Type":"ContainerDied","Data":"2174b8faf9196800c9ab9e036bfee4f7e04f44740d3bf5df4da9b69418ec3d07"}
Oct 30 23:00:04.215492 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.215474    4581 scope.go:122] "RemoveContainer" containerID="2174b8faf9196800c9ab9e036bfee4f7e04f44740d3bf5df4da9b69418ec3d07"
Oct 30 23:00:04.217110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.217088    4581 generic.go:356] "Generic (PLEG): container finished" podID="339358cd-ae97-49d2-a336-13be6d694007" containerID="0eacf17682d3ed649a4ebb94af20464fc1d4f7a405143fb8c429c637a22d5c41" exitCode=60
Oct 30 23:00:04.217197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.217167    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" event={"ID":"339358cd-ae97-49d2-a336-13be6d694007","Type":"ContainerDied","Data":"0eacf17682d3ed649a4ebb94af20464fc1d4f7a405143fb8c429c637a22d5c41"}
Oct 30 23:00:04.217399 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.217386    4581 scope.go:122] "RemoveContainer" containerID="0eacf17682d3ed649a4ebb94af20464fc1d4f7a405143fb8c429c637a22d5c41"
Oct 30 23:00:04.219676 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.219626    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" event={"ID":"db78128e-6cd0-445d-a667-075d7024ac7a","Type":"ContainerStarted","Data":"dd172193960e0a80dfa40ce383fdb6bac8494ad686615236c2f870274abf0cd9"}
Oct 30 23:00:04.219874 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:04.219858    4581 scope.go:122] "RemoveContainer" containerID="dd172193960e0a80dfa40ce383fdb6bac8494ad686615236c2f870274abf0cd9"
Oct 30 23:00:04.220085 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:04.220068    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq_odh-ai-gateway-infra(db78128e-6cd0-445d-a667-075d7024ac7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" podUID="db78128e-6cd0-445d-a667-075d7024ac7a"
Oct 30 23:00:05.234236 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.234186    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" event={"ID":"564d8a9f-e41c-41f5-8a03-90277087de3f","Type":"ContainerStarted","Data":"75670e39b927ebf58dac1c307e30b506cdcfd1b695cec1f94f7083b70de6477f"}
Oct 30 23:00:05.236843 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.236814    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" event={"ID":"314f0b61-51e0-463d-8ee3-5e2cfc934690","Type":"ContainerStarted","Data":"aa8a4fd041e021575f2c399139cbae14693445312bdfed9c92e61cd766a2d1e9"}
Oct 30 23:00:05.239011 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.238983    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" event={"ID":"70232370-49f1-43ba-a855-c3ed07d277b8","Type":"ContainerStarted","Data":"c58f21548cb5bc7711723508ffad328cd721f452ef1b3077681c56f9313c9f8e"}
Oct 30 23:00:05.241195 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.241174    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" event={"ID":"2a2d57df-edc4-4ffa-9308-4432b482085c","Type":"ContainerStarted","Data":"561992e101a6886a5ce1e048c150734a197183229597eac1fd2192ae284efae3"}
Oct 30 23:00:05.243642 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.243620    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" event={"ID":"e2671f5c-fcc0-46f8-8059-a8da58102f7b","Type":"ContainerStarted","Data":"33ca8b59741b148c075839aebbbac8c5996d558d88f3dc9b72e1a69025927612"}
Oct 30 23:00:05.245910 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.245879    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" event={"ID":"f084d37b-3a9e-41b3-8820-ffa93587d8be","Type":"ContainerStarted","Data":"dcd15d051158f525480be28b6528d4cb1827f7f6a87582cd3ae911d358d1f311"}
Oct 30 23:00:05.247609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.247589    4581 generic.go:356] "Generic (PLEG): container finished" podID="e3259e63-46a1-4572-9e46-b249d5fc24e8" containerID="584b797514918227bc688aefd445e6b4b0cfaab14d5f4275007b6cf1081d7c87" exitCode=60
Oct 30 23:00:05.247732 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.247669    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" event={"ID":"e3259e63-46a1-4572-9e46-b249d5fc24e8","Type":"ContainerDied","Data":"584b797514918227bc688aefd445e6b4b0cfaab14d5f4275007b6cf1081d7c87"}
Oct 30 23:00:05.247871 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.247855    4581 scope.go:122] "RemoveContainer" containerID="584b797514918227bc688aefd445e6b4b0cfaab14d5f4275007b6cf1081d7c87"
Oct 30 23:00:05.248148 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:05.248129    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw_odh-ai-gateway-infra(e3259e63-46a1-4572-9e46-b249d5fc24e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8"
Oct 30 23:00:05.249908 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.249886    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" event={"ID":"d1797edd-fa79-435c-9e29-3b93095377e1","Type":"ContainerStarted","Data":"3c8ca8f840c70dab4f3274671a36a9cf03c8b2c5741d1a2ca4431a985ab7745d"}
Oct 30 23:00:05.256103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.252525    4581 generic.go:356] "Generic (PLEG): container finished" podID="db78128e-6cd0-445d-a667-075d7024ac7a" containerID="dd172193960e0a80dfa40ce383fdb6bac8494ad686615236c2f870274abf0cd9" exitCode=60
Oct 30 23:00:05.256103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.252566    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" event={"ID":"db78128e-6cd0-445d-a667-075d7024ac7a","Type":"ContainerDied","Data":"dd172193960e0a80dfa40ce383fdb6bac8494ad686615236c2f870274abf0cd9"}
Oct 30 23:00:05.256103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.252605    4581 scope.go:122] "RemoveContainer" containerID="bf0f68e972c178bc97fd0fb9438c9613989fe457841fb247fed4c465f7c422d2"
Oct 30 23:00:05.256103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:05.253005    4581 scope.go:122] "RemoveContainer" containerID="dd172193960e0a80dfa40ce383fdb6bac8494ad686615236c2f870274abf0cd9"
Oct 30 23:00:05.256103 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:05.253253    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq_odh-ai-gateway-infra(db78128e-6cd0-445d-a667-075d7024ac7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" podUID="db78128e-6cd0-445d-a667-075d7024ac7a"
Oct 30 23:00:06.268153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.268060    4581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerID="9ad9225af9700caaf2c5e0020e266a5c207543812b4716c2d83a62ab4147498b" exitCode=60
Oct 30 23:00:06.268153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.268139    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4" event={"ID":"aaa9064d-b09a-4b48-b007-0cf8e3c1efac","Type":"ContainerDied","Data":"9ad9225af9700caaf2c5e0020e266a5c207543812b4716c2d83a62ab4147498b"}
Oct 30 23:00:06.268710 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.268208    4581 scope.go:122] "RemoveContainer" containerID="e7d65f89dd7c823b79d2cc7258820940a65f9248d858259610a1ed1cb46c6937"
Oct 30 23:00:06.268710 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.268564    4581 scope.go:122] "RemoveContainer" containerID="9ad9225af9700caaf2c5e0020e266a5c207543812b4716c2d83a62ab4147498b"
Oct 30 23:00:06.269041 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.268998    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4_odh-ai-gateway-infra(aaa9064d-b09a-4b48-b007-0cf8e3c1efac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac"
Oct 30 23:00:06.270536 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.270512    4581 generic.go:356] "Generic (PLEG): container finished" podID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerID="561992e101a6886a5ce1e048c150734a197183229597eac1fd2192ae284efae3" exitCode=60
Oct 30 23:00:06.270625 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.270594    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" event={"ID":"2a2d57df-edc4-4ffa-9308-4432b482085c","Type":"ContainerDied","Data":"561992e101a6886a5ce1e048c150734a197183229597eac1fd2192ae284efae3"}
Oct 30 23:00:06.270936 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.270915    4581 scope.go:122] "RemoveContainer" containerID="561992e101a6886a5ce1e048c150734a197183229597eac1fd2192ae284efae3"
Oct 30 23:00:06.271362 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.271336    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz_odh-ai-gateway-infra(2a2d57df-edc4-4ffa-9308-4432b482085c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c"
Oct 30 23:00:06.272852 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.272829    4581 generic.go:356] "Generic (PLEG): container finished" podID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerID="33ca8b59741b148c075839aebbbac8c5996d558d88f3dc9b72e1a69025927612" exitCode=60
Oct 30 23:00:06.272948 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.272913    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" event={"ID":"e2671f5c-fcc0-46f8-8059-a8da58102f7b","Type":"ContainerDied","Data":"33ca8b59741b148c075839aebbbac8c5996d558d88f3dc9b72e1a69025927612"}
Oct 30 23:00:06.273328 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.273175    4581 scope.go:122] "RemoveContainer" containerID="33ca8b59741b148c075839aebbbac8c5996d558d88f3dc9b72e1a69025927612"
Oct 30 23:00:06.273534 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.273513    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb_odh-ai-gateway-infra(e2671f5c-fcc0-46f8-8059-a8da58102f7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b"
Oct 30 23:00:06.275706 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.275672    4581 generic.go:356] "Generic (PLEG): container finished" podID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerID="dcd15d051158f525480be28b6528d4cb1827f7f6a87582cd3ae911d358d1f311" exitCode=60
Oct 30 23:00:06.275803 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.275759    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" event={"ID":"f084d37b-3a9e-41b3-8820-ffa93587d8be","Type":"ContainerDied","Data":"dcd15d051158f525480be28b6528d4cb1827f7f6a87582cd3ae911d358d1f311"}
Oct 30 23:00:06.276059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.276043    4581 scope.go:122] "RemoveContainer" containerID="dcd15d051158f525480be28b6528d4cb1827f7f6a87582cd3ae911d358d1f311"
Oct 30 23:00:06.276441 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.276417    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw_odh-ai-gateway-infra(f084d37b-3a9e-41b3-8820-ffa93587d8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be"
Oct 30 23:00:06.278089 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.278065    4581 generic.go:356] "Generic (PLEG): container finished" podID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerID="f17f4f575ed302ffe5671b3c66cc426ce1b5ee2fc78d6d50e9229c7e70590ee7" exitCode=60
Oct 30 23:00:06.278186 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.278145    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" event={"ID":"bb8e28dd-5a24-420b-8233-38b38e2bfda3","Type":"ContainerDied","Data":"f17f4f575ed302ffe5671b3c66cc426ce1b5ee2fc78d6d50e9229c7e70590ee7"}
Oct 30 23:00:06.278486 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.278471    4581 scope.go:122] "RemoveContainer" containerID="f17f4f575ed302ffe5671b3c66cc426ce1b5ee2fc78d6d50e9229c7e70590ee7"
Oct 30 23:00:06.278802 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.278782    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4_odh-ai-gateway-infra(bb8e28dd-5a24-420b-8233-38b38e2bfda3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3"
Oct 30 23:00:06.280734 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.280713    4581 generic.go:356] "Generic (PLEG): container finished" podID="d1797edd-fa79-435c-9e29-3b93095377e1" containerID="3c8ca8f840c70dab4f3274671a36a9cf03c8b2c5741d1a2ca4431a985ab7745d" exitCode=60
Oct 30 23:00:06.280823 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.280747    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" event={"ID":"d1797edd-fa79-435c-9e29-3b93095377e1","Type":"ContainerDied","Data":"3c8ca8f840c70dab4f3274671a36a9cf03c8b2c5741d1a2ca4431a985ab7745d"}
Oct 30 23:00:06.281078 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.281057    4581 scope.go:122] "RemoveContainer" containerID="3c8ca8f840c70dab4f3274671a36a9cf03c8b2c5741d1a2ca4431a985ab7745d"
Oct 30 23:00:06.281431 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.281403    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w_odh-ai-gateway-infra(d1797edd-fa79-435c-9e29-3b93095377e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" podUID="d1797edd-fa79-435c-9e29-3b93095377e1"
Oct 30 23:00:06.283074 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.283051    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" event={"ID":"36d2e83a-4650-437c-a8fb-422c4931413f","Type":"ContainerStarted","Data":"3586622d14b9fcc98ac15b436461ff1eb33057ea5ca3b3e9cd008eae9321732f"}
Oct 30 23:00:06.285421 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.285401    4581 generic.go:356] "Generic (PLEG): container finished" podID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerID="e3b4f23604fe2366949b012086677d97ab7ea86b814589450dfec4f16d8313f1" exitCode=60
Oct 30 23:00:06.285519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.285480    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf" event={"ID":"26267bf8-2af8-45fc-8640-6a0a3812ee2f","Type":"ContainerDied","Data":"e3b4f23604fe2366949b012086677d97ab7ea86b814589450dfec4f16d8313f1"}
Oct 30 23:00:06.285734 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.285720    4581 scope.go:122] "RemoveContainer" containerID="e3b4f23604fe2366949b012086677d97ab7ea86b814589450dfec4f16d8313f1"
Oct 30 23:00:06.286024 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.286001    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf_odh-ai-gateway-infra(26267bf8-2af8-45fc-8640-6a0a3812ee2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f"
Oct 30 23:00:06.287842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.287820    4581 generic.go:356] "Generic (PLEG): container finished" podID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerID="bc34a9fe2fb716c1cf1225c3c89abcfed76732abfc7e93c7e8b68a18322f3e17" exitCode=60
Oct 30 23:00:06.287953 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.287886    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" event={"ID":"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9","Type":"ContainerDied","Data":"bc34a9fe2fb716c1cf1225c3c89abcfed76732abfc7e93c7e8b68a18322f3e17"}
Oct 30 23:00:06.288160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.288142    4581 scope.go:122] "RemoveContainer" containerID="bc34a9fe2fb716c1cf1225c3c89abcfed76732abfc7e93c7e8b68a18322f3e17"
Oct 30 23:00:06.288479 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.288457    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c_odh-ai-gateway-infra(48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9"
Oct 30 23:00:06.289947 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.289911    4581 generic.go:356] "Generic (PLEG): container finished" podID="339358cd-ae97-49d2-a336-13be6d694007" containerID="40fc9a68ab196056b000ce49c98a1903bd86cca3dd18f3eeb58fc1a493d273af" exitCode=60
Oct 30 23:00:06.290039 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.289987    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" event={"ID":"339358cd-ae97-49d2-a336-13be6d694007","Type":"ContainerDied","Data":"40fc9a68ab196056b000ce49c98a1903bd86cca3dd18f3eeb58fc1a493d273af"}
Oct 30 23:00:06.290271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.290250    4581 scope.go:122] "RemoveContainer" containerID="40fc9a68ab196056b000ce49c98a1903bd86cca3dd18f3eeb58fc1a493d273af"
Oct 30 23:00:06.290546 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.290522    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7_odh-ai-gateway-infra(339358cd-ae97-49d2-a336-13be6d694007)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" podUID="339358cd-ae97-49d2-a336-13be6d694007"
Oct 30 23:00:06.292207 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.292189    4581 scope.go:122] "RemoveContainer" containerID="dd172193960e0a80dfa40ce383fdb6bac8494ad686615236c2f870274abf0cd9"
Oct 30 23:00:06.292454 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.292436    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq_odh-ai-gateway-infra(db78128e-6cd0-445d-a667-075d7024ac7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" podUID="db78128e-6cd0-445d-a667-075d7024ac7a"
Oct 30 23:00:06.294113 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.294092    4581 generic.go:356] "Generic (PLEG): container finished" podID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerID="eeeb3b4b5d4333a8e3db505a4fe7eca3b38b7b76b32d0c60bc78d77aecec3b7d" exitCode=60
Oct 30 23:00:06.294211 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.294173    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" event={"ID":"4663c26d-c795-4d52-9fdd-47b91bd76fe1","Type":"ContainerDied","Data":"eeeb3b4b5d4333a8e3db505a4fe7eca3b38b7b76b32d0c60bc78d77aecec3b7d"}
Oct 30 23:00:06.294422 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.294406    4581 scope.go:122] "RemoveContainer" containerID="eeeb3b4b5d4333a8e3db505a4fe7eca3b38b7b76b32d0c60bc78d77aecec3b7d"
Oct 30 23:00:06.294705 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.294667    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb_odh-ai-gateway-infra(4663c26d-c795-4d52-9fdd-47b91bd76fe1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1"
Oct 30 23:00:06.296137 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.296116    4581 generic.go:356] "Generic (PLEG): container finished" podID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerID="75670e39b927ebf58dac1c307e30b506cdcfd1b695cec1f94f7083b70de6477f" exitCode=60
Oct 30 23:00:06.296235 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.296189    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" event={"ID":"564d8a9f-e41c-41f5-8a03-90277087de3f","Type":"ContainerDied","Data":"75670e39b927ebf58dac1c307e30b506cdcfd1b695cec1f94f7083b70de6477f"}
Oct 30 23:00:06.296457 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.296428    4581 scope.go:122] "RemoveContainer" containerID="75670e39b927ebf58dac1c307e30b506cdcfd1b695cec1f94f7083b70de6477f"
Oct 30 23:00:06.296737 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.296711    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs_odh-ai-gateway-infra(564d8a9f-e41c-41f5-8a03-90277087de3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f"
Oct 30 23:00:06.298377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.298358    4581 generic.go:356] "Generic (PLEG): container finished" podID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerID="aa8a4fd041e021575f2c399139cbae14693445312bdfed9c92e61cd766a2d1e9" exitCode=60
Oct 30 23:00:06.298479 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.298390    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" event={"ID":"314f0b61-51e0-463d-8ee3-5e2cfc934690","Type":"ContainerDied","Data":"aa8a4fd041e021575f2c399139cbae14693445312bdfed9c92e61cd766a2d1e9"}
Oct 30 23:00:06.298680 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.298660    4581 scope.go:122] "RemoveContainer" containerID="aa8a4fd041e021575f2c399139cbae14693445312bdfed9c92e61cd766a2d1e9"
Oct 30 23:00:06.298955 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.298929    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn_odh-ai-gateway-infra(314f0b61-51e0-463d-8ee3-5e2cfc934690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690"
Oct 30 23:00:06.300474 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.300453    4581 generic.go:356] "Generic (PLEG): container finished" podID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerID="37646da3ef4d63f7f9ca0ab1136cff69364a31a7dbea52c79b10f0f0f7992d03" exitCode=60
Oct 30 23:00:06.300601 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.300484    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" event={"ID":"d7c80734-09bd-439c-baf5-95906d4a7bf0","Type":"ContainerDied","Data":"37646da3ef4d63f7f9ca0ab1136cff69364a31a7dbea52c79b10f0f0f7992d03"}
Oct 30 23:00:06.300788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.300773    4581 scope.go:122] "RemoveContainer" containerID="37646da3ef4d63f7f9ca0ab1136cff69364a31a7dbea52c79b10f0f0f7992d03"
Oct 30 23:00:06.301076 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.301051    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd_odh-ai-gateway-infra(d7c80734-09bd-439c-baf5-95906d4a7bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0"
Oct 30 23:00:06.302615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.302598    4581 generic.go:356] "Generic (PLEG): container finished" podID="70232370-49f1-43ba-a855-c3ed07d277b8" containerID="c58f21548cb5bc7711723508ffad328cd721f452ef1b3077681c56f9313c9f8e" exitCode=60
Oct 30 23:00:06.302698 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.302679    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" event={"ID":"70232370-49f1-43ba-a855-c3ed07d277b8","Type":"ContainerDied","Data":"c58f21548cb5bc7711723508ffad328cd721f452ef1b3077681c56f9313c9f8e"}
Oct 30 23:00:06.302857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.302841    4581 scope.go:122] "RemoveContainer" containerID="c58f21548cb5bc7711723508ffad328cd721f452ef1b3077681c56f9313c9f8e"
Oct 30 23:00:06.302938 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.302925    4581 scope.go:122] "RemoveContainer" containerID="584b797514918227bc688aefd445e6b4b0cfaab14d5f4275007b6cf1081d7c87"
Oct 30 23:00:06.303082 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.303051    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw_odh-ai-gateway-infra(70232370-49f1-43ba-a855-c3ed07d277b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" podUID="70232370-49f1-43ba-a855-c3ed07d277b8"
Oct 30 23:00:06.303171 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:06.303145    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw_odh-ai-gateway-infra(e3259e63-46a1-4572-9e46-b249d5fc24e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8"
Oct 30 23:00:06.410806 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.410709    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" podStartSLOduration=6.410694563 podStartE2EDuration="6.410694563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:06.409088881 +0000 UTC m=+3190.448027768" watchObservedRunningTime="2026-10-30 23:00:06.410694563 +0000 UTC m=+3190.449633449"
Oct 30 23:00:06.474477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.474450    4581 scope.go:122] "RemoveContainer" containerID="d2690122e7e98fa8e45da0eca22066103325c47e4a15b3ac3faa6d542cb20cb1"
Oct 30 23:00:06.491293 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.491262    4581 scope.go:122] "RemoveContainer" containerID="e301a63944b6d8b2f9aa9b3c98f756dfe7c1b8b296b4aec784acbc869b5b19ef"
Oct 30 23:00:06.507377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.507352    4581 scope.go:122] "RemoveContainer" containerID="e86dd0cbf64b73a4e67a5bd87805f08d33e641e728fd05262aa63d734b5ac36b"
Oct 30 23:00:06.522609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.522581    4581 scope.go:122] "RemoveContainer" containerID="629d845db89c7e386ad943602d9c8946df266ce78478a82a80a9fda2adcacaf5"
Oct 30 23:00:06.538104 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.538084    4581 scope.go:122] "RemoveContainer" containerID="c219713df36f302fd761b810def99897cfb3aead91304ae73d57e4f709caf7ab"
Oct 30 23:00:06.555061 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.555035    4581 scope.go:122] "RemoveContainer" containerID="e1a216a90b08c24d06bb01aa2980d0103ce9054eece706343f974540dbb66268"
Oct 30 23:00:06.588916 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.588889    4581 scope.go:122] "RemoveContainer" containerID="2174b8faf9196800c9ab9e036bfee4f7e04f44740d3bf5df4da9b69418ec3d07"
Oct 30 23:00:06.605934 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.605912    4581 scope.go:122] "RemoveContainer" containerID="0eacf17682d3ed649a4ebb94af20464fc1d4f7a405143fb8c429c637a22d5c41"
Oct 30 23:00:06.626088 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.626063    4581 scope.go:122] "RemoveContainer" containerID="a782739235a6ec58cdb93987479f1142902ab4fb59ea4c98e97de99adfb242d2"
Oct 30 23:00:06.641963 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.641935    4581 scope.go:122] "RemoveContainer" containerID="cafe5ccd2d7e40f82cfba27fb1fe5060d5b99e2912dff461a6e5476769c9c1b3"
Oct 30 23:00:06.659007 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.658984    4581 scope.go:122] "RemoveContainer" containerID="90d411ce2ae17ff0a8791cc62c54b6aed65ce93d1317b3cd274bd818fcd9716d"
Oct 30 23:00:06.675826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.675802    4581 scope.go:122] "RemoveContainer" containerID="b8e66c9c3b81bf53605a9fc0d217ac0674025f0a364cd26da57553b194c9f9b7"
Oct 30 23:00:06.694081 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:06.694057    4581 scope.go:122] "RemoveContainer" containerID="a6b70d4abad04c05df0bd7f55d2cd1471b0617b435782db790085247e45c3459"
Oct 30 23:00:07.313789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.313737    4581 scope.go:122] "RemoveContainer" containerID="3c8ca8f840c70dab4f3274671a36a9cf03c8b2c5741d1a2ca4431a985ab7745d"
Oct 30 23:00:07.314279 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.314179    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w_odh-ai-gateway-infra(d1797edd-fa79-435c-9e29-3b93095377e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" podUID="d1797edd-fa79-435c-9e29-3b93095377e1"
Oct 30 23:00:07.315530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.315507    4581 generic.go:356] "Generic (PLEG): container finished" podID="36d2e83a-4650-437c-a8fb-422c4931413f" containerID="3586622d14b9fcc98ac15b436461ff1eb33057ea5ca3b3e9cd008eae9321732f" exitCode=60
Oct 30 23:00:07.315624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.315602    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" event={"ID":"36d2e83a-4650-437c-a8fb-422c4931413f","Type":"ContainerDied","Data":"3586622d14b9fcc98ac15b436461ff1eb33057ea5ca3b3e9cd008eae9321732f"}
Oct 30 23:00:07.315714 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.315697    4581 scope.go:122] "RemoveContainer" containerID="4b0cd5bd043db0707d2fa75a506639bda66ef811d01f4dbba389fc286a2545e6"
Oct 30 23:00:07.315861 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.315843    4581 scope.go:122] "RemoveContainer" containerID="3586622d14b9fcc98ac15b436461ff1eb33057ea5ca3b3e9cd008eae9321732f"
Oct 30 23:00:07.316291 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.316262    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h_odh-ai-gateway-infra(36d2e83a-4650-437c-a8fb-422c4931413f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" podUID="36d2e83a-4650-437c-a8fb-422c4931413f"
Oct 30 23:00:07.317852 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.317836    4581 scope.go:122] "RemoveContainer" containerID="e3b4f23604fe2366949b012086677d97ab7ea86b814589450dfec4f16d8313f1"
Oct 30 23:00:07.318177 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.318151    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf_odh-ai-gateway-infra(26267bf8-2af8-45fc-8640-6a0a3812ee2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f"
Oct 30 23:00:07.319454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.319436    4581 scope.go:122] "RemoveContainer" containerID="bc34a9fe2fb716c1cf1225c3c89abcfed76732abfc7e93c7e8b68a18322f3e17"
Oct 30 23:00:07.319759 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.319738    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c_odh-ai-gateway-infra(48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9"
Oct 30 23:00:07.321110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.321095    4581 scope.go:122] "RemoveContainer" containerID="40fc9a68ab196056b000ce49c98a1903bd86cca3dd18f3eeb58fc1a493d273af"
Oct 30 23:00:07.321429 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.321404    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7_odh-ai-gateway-infra(339358cd-ae97-49d2-a336-13be6d694007)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" podUID="339358cd-ae97-49d2-a336-13be6d694007"
Oct 30 23:00:07.322980 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.322959    4581 scope.go:122] "RemoveContainer" containerID="eeeb3b4b5d4333a8e3db505a4fe7eca3b38b7b76b32d0c60bc78d77aecec3b7d"
Oct 30 23:00:07.323322 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.323293    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb_odh-ai-gateway-infra(4663c26d-c795-4d52-9fdd-47b91bd76fe1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1"
Oct 30 23:00:07.324551 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.324535    4581 scope.go:122] "RemoveContainer" containerID="75670e39b927ebf58dac1c307e30b506cdcfd1b695cec1f94f7083b70de6477f"
Oct 30 23:00:07.324803 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.324771    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs_odh-ai-gateway-infra(564d8a9f-e41c-41f5-8a03-90277087de3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f"
Oct 30 23:00:07.326420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.326402    4581 scope.go:122] "RemoveContainer" containerID="aa8a4fd041e021575f2c399139cbae14693445312bdfed9c92e61cd766a2d1e9"
Oct 30 23:00:07.326661 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.326640    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn_odh-ai-gateway-infra(314f0b61-51e0-463d-8ee3-5e2cfc934690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690"
Oct 30 23:00:07.328109 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.328092    4581 scope.go:122] "RemoveContainer" containerID="37646da3ef4d63f7f9ca0ab1136cff69364a31a7dbea52c79b10f0f0f7992d03"
Oct 30 23:00:07.328417 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.328395    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd_odh-ai-gateway-infra(d7c80734-09bd-439c-baf5-95906d4a7bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0"
Oct 30 23:00:07.329804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.329786    4581 scope.go:122] "RemoveContainer" containerID="c58f21548cb5bc7711723508ffad328cd721f452ef1b3077681c56f9313c9f8e"
Oct 30 23:00:07.330036 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.330011    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw_odh-ai-gateway-infra(70232370-49f1-43ba-a855-c3ed07d277b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" podUID="70232370-49f1-43ba-a855-c3ed07d277b8"
Oct 30 23:00:07.331531 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.331513    4581 scope.go:122] "RemoveContainer" containerID="9ad9225af9700caaf2c5e0020e266a5c207543812b4716c2d83a62ab4147498b"
Oct 30 23:00:07.331761 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.331736    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4_odh-ai-gateway-infra(aaa9064d-b09a-4b48-b007-0cf8e3c1efac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac"
Oct 30 23:00:07.333197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.333177    4581 scope.go:122] "RemoveContainer" containerID="561992e101a6886a5ce1e048c150734a197183229597eac1fd2192ae284efae3"
Oct 30 23:00:07.333440 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.333419    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz_odh-ai-gateway-infra(2a2d57df-edc4-4ffa-9308-4432b482085c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c"
Oct 30 23:00:07.334900 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.334876    4581 scope.go:122] "RemoveContainer" containerID="33ca8b59741b148c075839aebbbac8c5996d558d88f3dc9b72e1a69025927612"
Oct 30 23:00:07.335125 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.335099    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb_odh-ai-gateway-infra(e2671f5c-fcc0-46f8-8059-a8da58102f7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b"
Oct 30 23:00:07.336567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.336547    4581 scope.go:122] "RemoveContainer" containerID="dcd15d051158f525480be28b6528d4cb1827f7f6a87582cd3ae911d358d1f311"
Oct 30 23:00:07.336786 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.336763    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw_odh-ai-gateway-infra(f084d37b-3a9e-41b3-8820-ffa93587d8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be"
Oct 30 23:00:07.338135 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:07.338119    4581 scope.go:122] "RemoveContainer" containerID="f17f4f575ed302ffe5671b3c66cc426ce1b5ee2fc78d6d50e9229c7e70590ee7"
Oct 30 23:00:07.338356 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:07.338339    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4_odh-ai-gateway-infra(bb8e28dd-5a24-420b-8233-38b38e2bfda3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3"
Oct 30 23:00:08.350789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:08.350759    4581 scope.go:122] "RemoveContainer" containerID="3586622d14b9fcc98ac15b436461ff1eb33057ea5ca3b3e9cd008eae9321732f"
Oct 30 23:00:08.351303 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:08.351201    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h_odh-ai-gateway-infra(36d2e83a-4650-437c-a8fb-422c4931413f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" podUID="36d2e83a-4650-437c-a8fb-422c4931413f"
Oct 30 23:00:10.818852 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:10.818810    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:00:15.984087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:15.984055    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"]
Oct 30 23:00:16.138529 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.138319    4581 scope.go:122] "RemoveContainer" containerID="dcd15d051158f525480be28b6528d4cb1827f7f6a87582cd3ae911d358d1f311"
Oct 30 23:00:16.139132 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.138829    4581 scope.go:122] "RemoveContainer" containerID="dd172193960e0a80dfa40ce383fdb6bac8494ad686615236c2f870274abf0cd9"
Oct 30 23:00:16.231190 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.230776    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:16.298690 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.298651    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3259e63-46a1-4572-9e46-b249d5fc24e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3259e63-46a1-4572-9e46-b249d5fc24e8-maas-api-service-ca\") pod \"e3259e63-46a1-4572-9e46-b249d5fc24e8\" (UID: \"e3259e63-46a1-4572-9e46-b249d5fc24e8\") "
Oct 30 23:00:16.299021 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.298996    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3259e63-46a1-4572-9e46-b249d5fc24e8-kube-api-access-mb6bv\" (UniqueName: \"kubernetes.io/projected/e3259e63-46a1-4572-9e46-b249d5fc24e8-kube-api-access-mb6bv\") pod \"e3259e63-46a1-4572-9e46-b249d5fc24e8\" (UID: \"e3259e63-46a1-4572-9e46-b249d5fc24e8\") "
Oct 30 23:00:16.299308 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.299265    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3259e63-46a1-4572-9e46-b249d5fc24e8-maas-api-service-ca" pod "e3259e63-46a1-4572-9e46-b249d5fc24e8" (UID: "e3259e63-46a1-4572-9e46-b249d5fc24e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:16.299954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.299935    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3259e63-46a1-4572-9e46-b249d5fc24e8-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.301779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.301755    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3259e63-46a1-4572-9e46-b249d5fc24e8-kube-api-access-mb6bv" pod "e3259e63-46a1-4572-9e46-b249d5fc24e8" (UID: "e3259e63-46a1-4572-9e46-b249d5fc24e8"). InnerVolumeSpecName "kube-api-access-mb6bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:16.401420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.401385    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb6bv\" (UniqueName: \"kubernetes.io/projected/e3259e63-46a1-4572-9e46-b249d5fc24e8-kube-api-access-mb6bv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.438425 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.438391    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw" event={"ID":"e3259e63-46a1-4572-9e46-b249d5fc24e8","Type":"ContainerDied","Data":"90b3775e610d8eb468e3aa84b8d1a77fcc5b3188e7b4d984c994c75a0fb69ae7"}
Oct 30 23:00:16.438615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.438453    4581 scope.go:122] "RemoveContainer" containerID="584b797514918227bc688aefd445e6b4b0cfaab14d5f4275007b6cf1081d7c87"
Oct 30 23:00:16.438615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.438564    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"
Oct 30 23:00:16.531160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.531070    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"]
Oct 30 23:00:16.584031 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.583656    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e3a7dd-29890020-299zw"]
Oct 30 23:00:16.874641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.874618    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"]
Oct 30 23:00:16.882368 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:16.882342    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e3a7dd-fxlwn"]
Oct 30 23:00:17.089611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:17.089583    4581 scope.go:122] "RemoveContainer" containerID="bc34a9fe2fb716c1cf1225c3c89abcfed76732abfc7e93c7e8b68a18322f3e17"
Oct 30 23:00:17.090058 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:17.089734    4581 scope.go:122] "RemoveContainer" containerID="e3b4f23604fe2366949b012086677d97ab7ea86b814589450dfec4f16d8313f1"
Oct 30 23:00:17.090058 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:17.089758    4581 scope.go:122] "RemoveContainer" containerID="aa8a4fd041e021575f2c399139cbae14693445312bdfed9c92e61cd766a2d1e9"
Oct 30 23:00:17.449405 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:17.449371    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" event={"ID":"db78128e-6cd0-445d-a667-075d7024ac7a","Type":"ContainerStarted","Data":"128e607dfcb33e859d0a985ea05e546995858c97d3b69ef536329b688d8128ee"}
Oct 30 23:00:17.461048 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:17.461019    4581 generic.go:356] "Generic (PLEG): container finished" podID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerID="6c834f4d006d8b83b80c14d9b02a57a27b741281debf9f38babf02861eaac741" exitCode=60
Oct 30 23:00:17.461193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:17.461091    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" event={"ID":"f084d37b-3a9e-41b3-8820-ffa93587d8be","Type":"ContainerDied","Data":"6c834f4d006d8b83b80c14d9b02a57a27b741281debf9f38babf02861eaac741"}
Oct 30 23:00:17.461193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:17.461148    4581 scope.go:122] "RemoveContainer" containerID="dcd15d051158f525480be28b6528d4cb1827f7f6a87582cd3ae911d358d1f311"
Oct 30 23:00:17.461627 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:17.461614    4581 scope.go:122] "RemoveContainer" containerID="6c834f4d006d8b83b80c14d9b02a57a27b741281debf9f38babf02861eaac741"
Oct 30 23:00:17.462063 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:17.462026    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw_odh-ai-gateway-infra(f084d37b-3a9e-41b3-8820-ffa93587d8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be"
Oct 30 23:00:18.101069 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.101036    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d372c52d-b463-4e63-9df1-9c21d967e53c" path="/var/lib/kubelet/pods/d372c52d-b463-4e63-9df1-9c21d967e53c/volumes"
Oct 30 23:00:18.101631 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.101613    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8" path="/var/lib/kubelet/pods/e3259e63-46a1-4572-9e46-b249d5fc24e8/volumes"
Oct 30 23:00:18.103189 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.103150    4581 scope.go:122] "RemoveContainer" containerID="37646da3ef4d63f7f9ca0ab1136cff69364a31a7dbea52c79b10f0f0f7992d03"
Oct 30 23:00:18.103895 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.103748    4581 scope.go:122] "RemoveContainer" containerID="c58f21548cb5bc7711723508ffad328cd721f452ef1b3077681c56f9313c9f8e"
Oct 30 23:00:18.104505 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.103952    4581 scope.go:122] "RemoveContainer" containerID="75670e39b927ebf58dac1c307e30b506cdcfd1b695cec1f94f7083b70de6477f"
Oct 30 23:00:18.104505 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.104281    4581 scope.go:122] "RemoveContainer" containerID="eeeb3b4b5d4333a8e3db505a4fe7eca3b38b7b76b32d0c60bc78d77aecec3b7d"
Oct 30 23:00:18.106483 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.104669    4581 scope.go:122] "RemoveContainer" containerID="33ca8b59741b148c075839aebbbac8c5996d558d88f3dc9b72e1a69025927612"
Oct 30 23:00:18.474319 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.474288    4581 generic.go:356] "Generic (PLEG): container finished" podID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerID="43324a3e0d678da7fe4e21496ec6ceb0fa466d8a8bff204f4d2ce980239fcea2" exitCode=60
Oct 30 23:00:18.474495 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.474360    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf" event={"ID":"26267bf8-2af8-45fc-8640-6a0a3812ee2f","Type":"ContainerDied","Data":"43324a3e0d678da7fe4e21496ec6ceb0fa466d8a8bff204f4d2ce980239fcea2"}
Oct 30 23:00:18.474495 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.474437    4581 scope.go:122] "RemoveContainer" containerID="e3b4f23604fe2366949b012086677d97ab7ea86b814589450dfec4f16d8313f1"
Oct 30 23:00:18.474941 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.474917    4581 scope.go:122] "RemoveContainer" containerID="43324a3e0d678da7fe4e21496ec6ceb0fa466d8a8bff204f4d2ce980239fcea2"
Oct 30 23:00:18.475455 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:18.475413    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf_odh-ai-gateway-infra(26267bf8-2af8-45fc-8640-6a0a3812ee2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f"
Oct 30 23:00:18.476852 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.476827    4581 generic.go:356] "Generic (PLEG): container finished" podID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerID="e527ad8e01e9c275e0ede07f7f118a52251b845dac1d9079cb93e940158dea86" exitCode=60
Oct 30 23:00:18.476971 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.476910    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" event={"ID":"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9","Type":"ContainerDied","Data":"e527ad8e01e9c275e0ede07f7f118a52251b845dac1d9079cb93e940158dea86"}
Oct 30 23:00:18.477430 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.477407    4581 scope.go:122] "RemoveContainer" containerID="e527ad8e01e9c275e0ede07f7f118a52251b845dac1d9079cb93e940158dea86"
Oct 30 23:00:18.477837 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:18.477814    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c_odh-ai-gateway-infra(48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9"
Oct 30 23:00:18.479153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.479128    4581 generic.go:356] "Generic (PLEG): container finished" podID="db78128e-6cd0-445d-a667-075d7024ac7a" containerID="128e607dfcb33e859d0a985ea05e546995858c97d3b69ef536329b688d8128ee" exitCode=60
Oct 30 23:00:18.479318 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.479210    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" event={"ID":"db78128e-6cd0-445d-a667-075d7024ac7a","Type":"ContainerDied","Data":"128e607dfcb33e859d0a985ea05e546995858c97d3b69ef536329b688d8128ee"}
Oct 30 23:00:18.479666 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.479644    4581 scope.go:122] "RemoveContainer" containerID="128e607dfcb33e859d0a985ea05e546995858c97d3b69ef536329b688d8128ee"
Oct 30 23:00:18.480049 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:18.480022    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq_odh-ai-gateway-infra(db78128e-6cd0-445d-a667-075d7024ac7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" podUID="db78128e-6cd0-445d-a667-075d7024ac7a"
Oct 30 23:00:18.481569 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.481546    4581 generic.go:356] "Generic (PLEG): container finished" podID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerID="7eb7dea3f687872804f1d969741374f35f52784a3f313e656cbcbc9d16da4f0d" exitCode=60
Oct 30 23:00:18.481672 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.481619    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" event={"ID":"314f0b61-51e0-463d-8ee3-5e2cfc934690","Type":"ContainerDied","Data":"7eb7dea3f687872804f1d969741374f35f52784a3f313e656cbcbc9d16da4f0d"}
Oct 30 23:00:18.481990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.481973    4581 scope.go:122] "RemoveContainer" containerID="7eb7dea3f687872804f1d969741374f35f52784a3f313e656cbcbc9d16da4f0d"
Oct 30 23:00:18.482374 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:18.482305    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn_odh-ai-gateway-infra(314f0b61-51e0-463d-8ee3-5e2cfc934690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690"
Oct 30 23:00:18.490799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.490781    4581 scope.go:122] "RemoveContainer" containerID="bc34a9fe2fb716c1cf1225c3c89abcfed76732abfc7e93c7e8b68a18322f3e17"
Oct 30 23:00:18.507332 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.507302    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"]
Oct 30 23:00:18.510352 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.510331    4581 scope.go:122] "RemoveContainer" containerID="dd172193960e0a80dfa40ce383fdb6bac8494ad686615236c2f870274abf0cd9"
Oct 30 23:00:18.531407 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.531387    4581 scope.go:122] "RemoveContainer" containerID="aa8a4fd041e021575f2c399139cbae14693445312bdfed9c92e61cd766a2d1e9"
Oct 30 23:00:18.539926 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.539898    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"]
Oct 30 23:00:18.868469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.868439    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:18.984015 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.983985    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f084d37b-3a9e-41b3-8820-ffa93587d8be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084d37b-3a9e-41b3-8820-ffa93587d8be-maas-api-service-ca\") pod \"f084d37b-3a9e-41b3-8820-ffa93587d8be\" (UID: \"f084d37b-3a9e-41b3-8820-ffa93587d8be\") "
Oct 30 23:00:18.984179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.984111    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f084d37b-3a9e-41b3-8820-ffa93587d8be-kube-api-access-px8pl\" (UniqueName: \"kubernetes.io/projected/f084d37b-3a9e-41b3-8820-ffa93587d8be-kube-api-access-px8pl\") pod \"f084d37b-3a9e-41b3-8820-ffa93587d8be\" (UID: \"f084d37b-3a9e-41b3-8820-ffa93587d8be\") "
Oct 30 23:00:18.984676 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.984637    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f084d37b-3a9e-41b3-8820-ffa93587d8be-maas-api-service-ca" pod "f084d37b-3a9e-41b3-8820-ffa93587d8be" (UID: "f084d37b-3a9e-41b3-8820-ffa93587d8be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:18.984892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.984876    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f084d37b-3a9e-41b3-8820-ffa93587d8be-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:18.986506 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:18.986481    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f084d37b-3a9e-41b3-8820-ffa93587d8be-kube-api-access-px8pl" pod "f084d37b-3a9e-41b3-8820-ffa93587d8be" (UID: "f084d37b-3a9e-41b3-8820-ffa93587d8be"). InnerVolumeSpecName "kube-api-access-px8pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:19.087684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.087529    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px8pl\" (UniqueName: \"kubernetes.io/projected/f084d37b-3a9e-41b3-8820-ffa93587d8be-kube-api-access-px8pl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:19.088690 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.088651    4581 scope.go:122] "RemoveContainer" containerID="561992e101a6886a5ce1e048c150734a197183229597eac1fd2192ae284efae3"
Oct 30 23:00:19.090121 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.090096    4581 scope.go:122] "RemoveContainer" containerID="3586622d14b9fcc98ac15b436461ff1eb33057ea5ca3b3e9cd008eae9321732f"
Oct 30 23:00:19.091427 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.091390    4581 scope.go:122] "RemoveContainer" containerID="40fc9a68ab196056b000ce49c98a1903bd86cca3dd18f3eeb58fc1a493d273af"
Oct 30 23:00:19.529004 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.528962    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"]
Oct 30 23:00:19.536344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.532747    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" event={"ID":"4663c26d-c795-4d52-9fdd-47b91bd76fe1","Type":"ContainerStarted","Data":"bae9b1c35eb766b40134bdf16b3c6574103c0de57ab6bc87c8a8d78f604896ff"}
Oct 30 23:00:19.539362 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.539336    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" event={"ID":"564d8a9f-e41c-41f5-8a03-90277087de3f","Type":"ContainerStarted","Data":"b15ce24b117c2368f6e6a94c1649a914ac153840a589e1186ce79fb40cbe6536"}
Oct 30 23:00:19.568138 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.568095    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" event={"ID":"d7c80734-09bd-439c-baf5-95906d4a7bf0","Type":"ContainerStarted","Data":"eede7678d8810d769e06e7d30e34908ae15b706d9f6c0a884fb0e5f4b01b239a"}
Oct 30 23:00:19.571911 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.571875    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" event={"ID":"70232370-49f1-43ba-a855-c3ed07d277b8","Type":"ContainerStarted","Data":"4e840a10f25c6bb280f5f5ccc16c1b0c6ec3d4878ba258b7ddabf9f1f92dbf4d"}
Oct 30 23:00:19.576969 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.576936    4581 generic.go:356] "Generic (PLEG): container finished" podID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerID="b35f7d1cda5489365f2953171b064767f155ca1987fdb944d070bef6d0570b36" exitCode=60
Oct 30 23:00:19.578695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.578639    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" event={"ID":"e2671f5c-fcc0-46f8-8059-a8da58102f7b","Type":"ContainerDied","Data":"b35f7d1cda5489365f2953171b064767f155ca1987fdb944d070bef6d0570b36"}
Oct 30 23:00:19.580071 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.580051    4581 scope.go:122] "RemoveContainer" containerID="33ca8b59741b148c075839aebbbac8c5996d558d88f3dc9b72e1a69025927612"
Oct 30 23:00:19.580484 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.580458    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw" event={"ID":"f084d37b-3a9e-41b3-8820-ffa93587d8be","Type":"ContainerDied","Data":"46ea04aeda8ec49654ebe4a7b5686d083adbfbe43b6bbf3e58dcd49e1b3f038c"}
Oct 30 23:00:19.580645 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.580630    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"
Oct 30 23:00:19.580754 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.580731    4581 scope.go:122] "RemoveContainer" containerID="b35f7d1cda5489365f2953171b064767f155ca1987fdb944d070bef6d0570b36"
Oct 30 23:00:19.581181 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:19.581151    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb_odh-ai-gateway-infra(e2671f5c-fcc0-46f8-8059-a8da58102f7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b"
Oct 30 23:00:19.616327 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.616299    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"]
Oct 30 23:00:19.647256 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.646958    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"]
Oct 30 23:00:19.729853 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.729777    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" podStartSLOduration=19.729758565 podStartE2EDuration="19.729758565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:19.724113269 +0000 UTC m=+3203.763052154" watchObservedRunningTime="2026-10-30 23:00:19.729758565 +0000 UTC m=+3203.768697450"
Oct 30 23:00:19.865704 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.865678    4581 scope.go:122] "RemoveContainer" containerID="6c834f4d006d8b83b80c14d9b02a57a27b741281debf9f38babf02861eaac741"
Oct 30 23:00:19.954541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.954236    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"]
Oct 30 23:00:19.970241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.962523    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0a807-29890020-d4fkw"]
Oct 30 23:00:19.970241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:19.962963    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:20.059807 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.059769    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:20.065413 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.065380    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:20.125140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.125105    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" path="/var/lib/kubelet/pods/f084d37b-3a9e-41b3-8820-ffa93587d8be/volumes"
Oct 30 23:00:20.139402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.139143    4581 scope.go:122] "RemoveContainer" containerID="9ad9225af9700caaf2c5e0020e266a5c207543812b4716c2d83a62ab4147498b"
Oct 30 23:00:20.141125 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.140366    4581 scope.go:122] "RemoveContainer" containerID="f17f4f575ed302ffe5671b3c66cc426ce1b5ee2fc78d6d50e9229c7e70590ee7"
Oct 30 23:00:20.156256 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.154334    4581 scope.go:122] "RemoveContainer" containerID="3c8ca8f840c70dab4f3274671a36a9cf03c8b2c5741d1a2ca4431a985ab7745d"
Oct 30 23:00:20.156256 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.155071    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314f0b61-51e0-463d-8ee3-5e2cfc934690-kube-api-access-b6ckl\" (UniqueName: \"kubernetes.io/projected/314f0b61-51e0-463d-8ee3-5e2cfc934690-kube-api-access-b6ckl\") pod \"314f0b61-51e0-463d-8ee3-5e2cfc934690\" (UID: \"314f0b61-51e0-463d-8ee3-5e2cfc934690\") "
Oct 30 23:00:20.156483 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.156423    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26267bf8-2af8-45fc-8640-6a0a3812ee2f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26267bf8-2af8-45fc-8640-6a0a3812ee2f-maas-api-service-ca\") pod \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\" (UID: \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\") "
Oct 30 23:00:20.157166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.156649    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db78128e-6cd0-445d-a667-075d7024ac7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db78128e-6cd0-445d-a667-075d7024ac7a-maas-api-service-ca\") pod \"db78128e-6cd0-445d-a667-075d7024ac7a\" (UID: \"db78128e-6cd0-445d-a667-075d7024ac7a\") "
Oct 30 23:00:20.157166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.156696    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314f0b61-51e0-463d-8ee3-5e2cfc934690-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314f0b61-51e0-463d-8ee3-5e2cfc934690-maas-api-service-ca\") pod \"314f0b61-51e0-463d-8ee3-5e2cfc934690\" (UID: \"314f0b61-51e0-463d-8ee3-5e2cfc934690\") "
Oct 30 23:00:20.157166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.156730    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db78128e-6cd0-445d-a667-075d7024ac7a-kube-api-access-h8zh7\" (UniqueName: \"kubernetes.io/projected/db78128e-6cd0-445d-a667-075d7024ac7a-kube-api-access-h8zh7\") pod \"db78128e-6cd0-445d-a667-075d7024ac7a\" (UID: \"db78128e-6cd0-445d-a667-075d7024ac7a\") "
Oct 30 23:00:20.157166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.156949    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26267bf8-2af8-45fc-8640-6a0a3812ee2f-maas-api-service-ca" pod "26267bf8-2af8-45fc-8640-6a0a3812ee2f" (UID: "26267bf8-2af8-45fc-8640-6a0a3812ee2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:20.158133 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.157630    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db78128e-6cd0-445d-a667-075d7024ac7a-maas-api-service-ca" pod "db78128e-6cd0-445d-a667-075d7024ac7a" (UID: "db78128e-6cd0-445d-a667-075d7024ac7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:20.158133 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.157891    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db78128e-6cd0-445d-a667-075d7024ac7a-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.158133 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.157913    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26267bf8-2af8-45fc-8640-6a0a3812ee2f-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.158423 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.158254    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314f0b61-51e0-463d-8ee3-5e2cfc934690-maas-api-service-ca" pod "314f0b61-51e0-463d-8ee3-5e2cfc934690" (UID: "314f0b61-51e0-463d-8ee3-5e2cfc934690"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:20.168714 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.168683    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314f0b61-51e0-463d-8ee3-5e2cfc934690-kube-api-access-b6ckl" pod "314f0b61-51e0-463d-8ee3-5e2cfc934690" (UID: "314f0b61-51e0-463d-8ee3-5e2cfc934690"). InnerVolumeSpecName "kube-api-access-b6ckl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:20.168921 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.168875    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db78128e-6cd0-445d-a667-075d7024ac7a-kube-api-access-h8zh7" pod "db78128e-6cd0-445d-a667-075d7024ac7a" (UID: "db78128e-6cd0-445d-a667-075d7024ac7a"). InnerVolumeSpecName "kube-api-access-h8zh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:20.259741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.258608    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26267bf8-2af8-45fc-8640-6a0a3812ee2f-kube-api-access-vfcfh\" (UniqueName: \"kubernetes.io/projected/26267bf8-2af8-45fc-8640-6a0a3812ee2f-kube-api-access-vfcfh\") pod \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\" (UID: \"26267bf8-2af8-45fc-8640-6a0a3812ee2f\") "
Oct 30 23:00:20.259741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.259347    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314f0b61-51e0-463d-8ee3-5e2cfc934690-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.259741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.259369    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zh7\" (UniqueName: \"kubernetes.io/projected/db78128e-6cd0-445d-a667-075d7024ac7a-kube-api-access-h8zh7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.259741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.259384    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6ckl\" (UniqueName: \"kubernetes.io/projected/314f0b61-51e0-463d-8ee3-5e2cfc934690-kube-api-access-b6ckl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.283281 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.282130    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26267bf8-2af8-45fc-8640-6a0a3812ee2f-kube-api-access-vfcfh" pod "26267bf8-2af8-45fc-8640-6a0a3812ee2f" (UID: "26267bf8-2af8-45fc-8640-6a0a3812ee2f"). InnerVolumeSpecName "kube-api-access-vfcfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:20.304347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.304318    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:20.363420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.363386    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-kube-api-access-4w4vw\" (UniqueName: \"kubernetes.io/projected/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-kube-api-access-4w4vw\") pod \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\" (UID: \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\") "
Oct 30 23:00:20.363583 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.363557    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-maas-api-service-ca\") pod \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\" (UID: \"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9\") "
Oct 30 23:00:20.364287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.364208    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-maas-api-service-ca" pod "48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" (UID: "48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:20.368778 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.368748    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfcfh\" (UniqueName: \"kubernetes.io/projected/26267bf8-2af8-45fc-8640-6a0a3812ee2f-kube-api-access-vfcfh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.368915 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.368782    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.456249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.456108    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-kube-api-access-4w4vw" pod "48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" (UID: "48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9"). InnerVolumeSpecName "kube-api-access-4w4vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:20.470775 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.470738    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4vw\" (UniqueName: \"kubernetes.io/projected/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9-kube-api-access-4w4vw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.586803 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.586763    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"]
Oct 30 23:00:20.609105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.609071    4581 generic.go:356] "Generic (PLEG): container finished" podID="36d2e83a-4650-437c-a8fb-422c4931413f" containerID="07b31218336e73bff624ae35cce715a274bed774f1b738de987baf1ff3bd42e1" exitCode=60
Oct 30 23:00:20.609308 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.609143    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" event={"ID":"36d2e83a-4650-437c-a8fb-422c4931413f","Type":"ContainerDied","Data":"07b31218336e73bff624ae35cce715a274bed774f1b738de987baf1ff3bd42e1"}
Oct 30 23:00:20.609308 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.609229    4581 scope.go:122] "RemoveContainer" containerID="3586622d14b9fcc98ac15b436461ff1eb33057ea5ca3b3e9cd008eae9321732f"
Oct 30 23:00:20.609730 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.609704    4581 scope.go:122] "RemoveContainer" containerID="07b31218336e73bff624ae35cce715a274bed774f1b738de987baf1ff3bd42e1"
Oct 30 23:00:20.610163 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:20.610131    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h_odh-ai-gateway-infra(36d2e83a-4650-437c-a8fb-422c4931413f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" podUID="36d2e83a-4650-437c-a8fb-422c4931413f"
Oct 30 23:00:20.612380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.612360    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"
Oct 30 23:00:20.612624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.612369    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf" event={"ID":"26267bf8-2af8-45fc-8640-6a0a3812ee2f","Type":"ContainerDied","Data":"d2a9ab6f24f5241ce585be40840284748b48e35df84a908c7e4a139312f5755e"}
Oct 30 23:00:20.614397 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.614379    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"
Oct 30 23:00:20.617931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.617903    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c" event={"ID":"48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9","Type":"ContainerDied","Data":"bbe8bfa3e3ec521deda2837d922570b39646a54af1bd2d1e42c8430441f2dd11"}
Oct 30 23:00:20.629533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.629499    4581 generic.go:356] "Generic (PLEG): container finished" podID="339358cd-ae97-49d2-a336-13be6d694007" containerID="e7ba340546cd2e94820410f86f774a9b3da60398e95c8527f6ea8108955718b9" exitCode=60
Oct 30 23:00:20.629673 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.629614    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" event={"ID":"339358cd-ae97-49d2-a336-13be6d694007","Type":"ContainerDied","Data":"e7ba340546cd2e94820410f86f774a9b3da60398e95c8527f6ea8108955718b9"}
Oct 30 23:00:20.630153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.630129    4581 scope.go:122] "RemoveContainer" containerID="e7ba340546cd2e94820410f86f774a9b3da60398e95c8527f6ea8108955718b9"
Oct 30 23:00:20.631671 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:20.631641    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7_odh-ai-gateway-infra(339358cd-ae97-49d2-a336-13be6d694007)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" podUID="339358cd-ae97-49d2-a336-13be6d694007"
Oct 30 23:00:20.632901 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.632886    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"
Oct 30 23:00:20.632972 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.632925    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq" event={"ID":"db78128e-6cd0-445d-a667-075d7024ac7a","Type":"ContainerDied","Data":"a37b5c2619ffeba1d74ecd5cc29d954dda19504fff8b2d1f28b6320cb453325f"}
Oct 30 23:00:20.632972 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.632948    4581 scope.go:122] "RemoveContainer" containerID="43324a3e0d678da7fe4e21496ec6ceb0fa466d8a8bff204f4d2ce980239fcea2"
Oct 30 23:00:20.635748 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.635721    4581 generic.go:356] "Generic (PLEG): container finished" podID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerID="bae9b1c35eb766b40134bdf16b3c6574103c0de57ab6bc87c8a8d78f604896ff" exitCode=60
Oct 30 23:00:20.635851 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.635803    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" event={"ID":"4663c26d-c795-4d52-9fdd-47b91bd76fe1","Type":"ContainerDied","Data":"bae9b1c35eb766b40134bdf16b3c6574103c0de57ab6bc87c8a8d78f604896ff"}
Oct 30 23:00:20.639957 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.639934    4581 generic.go:356] "Generic (PLEG): container finished" podID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerID="b15ce24b117c2368f6e6a94c1649a914ac153840a589e1186ce79fb40cbe6536" exitCode=60
Oct 30 23:00:20.640248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.640198    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" event={"ID":"564d8a9f-e41c-41f5-8a03-90277087de3f","Type":"ContainerDied","Data":"b15ce24b117c2368f6e6a94c1649a914ac153840a589e1186ce79fb40cbe6536"}
Oct 30 23:00:20.640989 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.640972    4581 scope.go:122] "RemoveContainer" containerID="b15ce24b117c2368f6e6a94c1649a914ac153840a589e1186ce79fb40cbe6536"
Oct 30 23:00:20.641719 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:20.641697    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs_odh-ai-gateway-infra(564d8a9f-e41c-41f5-8a03-90277087de3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f"
Oct 30 23:00:20.642279 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.642256    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"]
Oct 30 23:00:20.646590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.646574    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"
Oct 30 23:00:20.646797 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.646598    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn" event={"ID":"314f0b61-51e0-463d-8ee3-5e2cfc934690","Type":"ContainerDied","Data":"3e0dcb575acb1e6567c365464a3bef1c2d60f6f74022aeb44be4698a7e8fbcde"}
Oct 30 23:00:20.652884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.652861    4581 generic.go:356] "Generic (PLEG): container finished" podID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerID="eede7678d8810d769e06e7d30e34908ae15b706d9f6c0a884fb0e5f4b01b239a" exitCode=60
Oct 30 23:00:20.653094 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.652888    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" event={"ID":"d7c80734-09bd-439c-baf5-95906d4a7bf0","Type":"ContainerDied","Data":"eede7678d8810d769e06e7d30e34908ae15b706d9f6c0a884fb0e5f4b01b239a"}
Oct 30 23:00:20.653745 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.653694    4581 scope.go:122] "RemoveContainer" containerID="eede7678d8810d769e06e7d30e34908ae15b706d9f6c0a884fb0e5f4b01b239a"
Oct 30 23:00:20.654942 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:20.654913    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd_odh-ai-gateway-infra(d7c80734-09bd-439c-baf5-95906d4a7bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0"
Oct 30 23:00:20.656161 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.656143    4581 scope.go:122] "RemoveContainer" containerID="e527ad8e01e9c275e0ede07f7f118a52251b845dac1d9079cb93e940158dea86"
Oct 30 23:00:20.658892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.658868    4581 generic.go:356] "Generic (PLEG): container finished" podID="70232370-49f1-43ba-a855-c3ed07d277b8" containerID="4e840a10f25c6bb280f5f5ccc16c1b0c6ec3d4878ba258b7ddabf9f1f92dbf4d" exitCode=60
Oct 30 23:00:20.659864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.659052    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" event={"ID":"70232370-49f1-43ba-a855-c3ed07d277b8","Type":"ContainerDied","Data":"4e840a10f25c6bb280f5f5ccc16c1b0c6ec3d4878ba258b7ddabf9f1f92dbf4d"}
Oct 30 23:00:20.662425 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.661585    4581 scope.go:122] "RemoveContainer" containerID="4e840a10f25c6bb280f5f5ccc16c1b0c6ec3d4878ba258b7ddabf9f1f92dbf4d"
Oct 30 23:00:20.662425 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:20.661902    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw_odh-ai-gateway-infra(70232370-49f1-43ba-a855-c3ed07d277b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" podUID="70232370-49f1-43ba-a855-c3ed07d277b8"
Oct 30 23:00:20.668460 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.668402    4581 generic.go:356] "Generic (PLEG): container finished" podID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerID="40133ec74bde6cf70e26846d51f3ce693269b23f5d89d4dc463d40f1b64c5ae1" exitCode=60
Oct 30 23:00:20.668782 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.668763    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" event={"ID":"2a2d57df-edc4-4ffa-9308-4432b482085c","Type":"ContainerDied","Data":"40133ec74bde6cf70e26846d51f3ce693269b23f5d89d4dc463d40f1b64c5ae1"}
Oct 30 23:00:20.672478 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.672460    4581 scope.go:122] "RemoveContainer" containerID="40133ec74bde6cf70e26846d51f3ce693269b23f5d89d4dc463d40f1b64c5ae1"
Oct 30 23:00:20.680323 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:20.680299    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz_odh-ai-gateway-infra(2a2d57df-edc4-4ffa-9308-4432b482085c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c"
Oct 30 23:00:20.686386 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.686353    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"]
Oct 30 23:00:20.691497 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.691475    4581 scope.go:122] "RemoveContainer" containerID="40fc9a68ab196056b000ce49c98a1903bd86cca3dd18f3eeb58fc1a493d273af"
Oct 30 23:00:20.713452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.713397    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"]
Oct 30 23:00:20.781169 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.781100    4581 scope.go:122] "RemoveContainer" containerID="128e607dfcb33e859d0a985ea05e546995858c97d3b69ef536329b688d8128ee"
Oct 30 23:00:20.794990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.794901    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"]
Oct 30 23:00:20.858027 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.855966    4581 scope.go:122] "RemoveContainer" containerID="eeeb3b4b5d4333a8e3db505a4fe7eca3b38b7b76b32d0c60bc78d77aecec3b7d"
Oct 30 23:00:20.869147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.869093    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"]
Oct 30 23:00:20.880793 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.880757    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a60e80-29890020-pt7nq"]
Oct 30 23:00:20.904270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.901301    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"]
Oct 30 23:00:20.907653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.906322    4581 scope.go:122] "RemoveContainer" containerID="75670e39b927ebf58dac1c307e30b506cdcfd1b695cec1f94f7083b70de6477f"
Oct 30 23:00:20.907653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.907466    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-23aab2-29890020-5tx8c"]
Oct 30 23:00:20.936380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.936354    4581 scope.go:122] "RemoveContainer" containerID="7eb7dea3f687872804f1d969741374f35f52784a3f313e656cbcbc9d16da4f0d"
Oct 30 23:00:20.942324 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.942207    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:20.970148 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.960086    4581 scope.go:122] "RemoveContainer" containerID="37646da3ef4d63f7f9ca0ab1136cff69364a31a7dbea52c79b10f0f0f7992d03"
Oct 30 23:00:20.980253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.979391    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4663c26d-c795-4d52-9fdd-47b91bd76fe1-kube-api-access-rlmll\" (UniqueName: \"kubernetes.io/projected/4663c26d-c795-4d52-9fdd-47b91bd76fe1-kube-api-access-rlmll\") pod \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\" (UID: \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\") "
Oct 30 23:00:20.980253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.979569    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4663c26d-c795-4d52-9fdd-47b91bd76fe1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4663c26d-c795-4d52-9fdd-47b91bd76fe1-maas-api-service-ca\") pod \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\" (UID: \"4663c26d-c795-4d52-9fdd-47b91bd76fe1\") "
Oct 30 23:00:20.980986 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.980757    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4663c26d-c795-4d52-9fdd-47b91bd76fe1-maas-api-service-ca" pod "4663c26d-c795-4d52-9fdd-47b91bd76fe1" (UID: "4663c26d-c795-4d52-9fdd-47b91bd76fe1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:20.993044 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.992314    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4663c26d-c795-4d52-9fdd-47b91bd76fe1-kube-api-access-rlmll" pod "4663c26d-c795-4d52-9fdd-47b91bd76fe1" (UID: "4663c26d-c795-4d52-9fdd-47b91bd76fe1"). InnerVolumeSpecName "kube-api-access-rlmll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:20.993398 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:20.993376    4581 scope.go:122] "RemoveContainer" containerID="c58f21548cb5bc7711723508ffad328cd721f452ef1b3077681c56f9313c9f8e"
Oct 30 23:00:21.020684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.020655    4581 scope.go:122] "RemoveContainer" containerID="561992e101a6886a5ce1e048c150734a197183229597eac1fd2192ae284efae3"
Oct 30 23:00:21.034741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.034705    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"]
Oct 30 23:00:21.045169 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.045119    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6f2eba-29890020-9kzmf"]
Oct 30 23:00:21.081863 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.081832    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4663c26d-c795-4d52-9fdd-47b91bd76fe1-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:21.083379 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.083356    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlmll\" (UniqueName: \"kubernetes.io/projected/4663c26d-c795-4d52-9fdd-47b91bd76fe1-kube-api-access-rlmll\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:21.089976 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.089953    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:21.183881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.183844    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"]
Oct 30 23:00:21.184617 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.184591    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2671f5c-fcc0-46f8-8059-a8da58102f7b-kube-api-access-hfmpz\" (UniqueName: \"kubernetes.io/projected/e2671f5c-fcc0-46f8-8059-a8da58102f7b-kube-api-access-hfmpz\") pod \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\" (UID: \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\") "
Oct 30 23:00:21.185020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.184881    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2671f5c-fcc0-46f8-8059-a8da58102f7b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2671f5c-fcc0-46f8-8059-a8da58102f7b-maas-api-service-ca\") pod \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\" (UID: \"e2671f5c-fcc0-46f8-8059-a8da58102f7b\") "
Oct 30 23:00:21.196312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.195425    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2671f5c-fcc0-46f8-8059-a8da58102f7b-kube-api-access-hfmpz" pod "e2671f5c-fcc0-46f8-8059-a8da58102f7b" (UID: "e2671f5c-fcc0-46f8-8059-a8da58102f7b"). InnerVolumeSpecName "kube-api-access-hfmpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:21.228419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.212783    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-09d0d5-29890020-k27jn"]
Oct 30 23:00:21.228419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.213851    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2671f5c-fcc0-46f8-8059-a8da58102f7b-maas-api-service-ca" pod "e2671f5c-fcc0-46f8-8059-a8da58102f7b" (UID: "e2671f5c-fcc0-46f8-8059-a8da58102f7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:21.287170 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.287137    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmpz\" (UniqueName: \"kubernetes.io/projected/e2671f5c-fcc0-46f8-8059-a8da58102f7b-kube-api-access-hfmpz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:21.287170 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.287166    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2671f5c-fcc0-46f8-8059-a8da58102f7b-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:21.677962 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.677914    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"]
Oct 30 23:00:21.692633 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.692604    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"
Oct 30 23:00:21.692633 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.692613    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb" event={"ID":"e2671f5c-fcc0-46f8-8059-a8da58102f7b","Type":"ContainerDied","Data":"9e1c386589f5d932a48dc6e13b99785faf7294a5b9254ebe9014927e7ff46103"}
Oct 30 23:00:21.692884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.692695    4581 scope.go:122] "RemoveContainer" containerID="b35f7d1cda5489365f2953171b064767f155ca1987fdb944d070bef6d0570b36"
Oct 30 23:00:21.695882 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.695851    4581 generic.go:356] "Generic (PLEG): container finished" podID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerID="62082a5fdb911ffcc2c0e9b6651e3ff1f9aba71b183c59a22f76ea728ad45eb7" exitCode=60
Oct 30 23:00:21.696025 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.695982    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" event={"ID":"bb8e28dd-5a24-420b-8233-38b38e2bfda3","Type":"ContainerDied","Data":"62082a5fdb911ffcc2c0e9b6651e3ff1f9aba71b183c59a22f76ea728ad45eb7"}
Oct 30 23:00:21.696402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.696383    4581 scope.go:122] "RemoveContainer" containerID="62082a5fdb911ffcc2c0e9b6651e3ff1f9aba71b183c59a22f76ea728ad45eb7"
Oct 30 23:00:21.696670 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:21.696649    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4_odh-ai-gateway-infra(bb8e28dd-5a24-420b-8233-38b38e2bfda3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3"
Oct 30 23:00:21.699938 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.699918    4581 generic.go:356] "Generic (PLEG): container finished" podID="d1797edd-fa79-435c-9e29-3b93095377e1" containerID="2590fe7d761d5b7e4f07a41542128cdd231d815f55b3d4f970833b64b12b73e4" exitCode=60
Oct 30 23:00:21.700044 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.700001    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" event={"ID":"d1797edd-fa79-435c-9e29-3b93095377e1","Type":"ContainerDied","Data":"2590fe7d761d5b7e4f07a41542128cdd231d815f55b3d4f970833b64b12b73e4"}
Oct 30 23:00:21.700420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.700400    4581 scope.go:122] "RemoveContainer" containerID="2590fe7d761d5b7e4f07a41542128cdd231d815f55b3d4f970833b64b12b73e4"
Oct 30 23:00:21.700774 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:21.700666    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w_odh-ai-gateway-infra(d1797edd-fa79-435c-9e29-3b93095377e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" podUID="d1797edd-fa79-435c-9e29-3b93095377e1"
Oct 30 23:00:21.717658 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.717423    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb" event={"ID":"4663c26d-c795-4d52-9fdd-47b91bd76fe1","Type":"ContainerDied","Data":"0f85897c95a50fd85676edb389b7f65b33268b9cf35b6c8d7e6716f223bc6db2"}
Oct 30 23:00:21.721327 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.718709    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"
Oct 30 23:00:21.721327 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.720723    4581 scope.go:122] "RemoveContainer" containerID="f17f4f575ed302ffe5671b3c66cc426ce1b5ee2fc78d6d50e9229c7e70590ee7"
Oct 30 23:00:21.738450 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.737062    4581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerID="8d414bb66524bfb90aa3459903d174eeb393fe8ac309a4d7584bda9a1f4f41e0" exitCode=60
Oct 30 23:00:21.738450 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.737161    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4" event={"ID":"aaa9064d-b09a-4b48-b007-0cf8e3c1efac","Type":"ContainerDied","Data":"8d414bb66524bfb90aa3459903d174eeb393fe8ac309a4d7584bda9a1f4f41e0"}
Oct 30 23:00:21.738450 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.737662    4581 scope.go:122] "RemoveContainer" containerID="8d414bb66524bfb90aa3459903d174eeb393fe8ac309a4d7584bda9a1f4f41e0"
Oct 30 23:00:21.738450 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:21.738080    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4_odh-ai-gateway-infra(aaa9064d-b09a-4b48-b007-0cf8e3c1efac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac"
Oct 30 23:00:21.782116 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.782085    4581 scope.go:122] "RemoveContainer" containerID="3c8ca8f840c70dab4f3274671a36a9cf03c8b2c5741d1a2ca4431a985ab7745d"
Oct 30 23:00:21.822488 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.819398    4581 scope.go:122] "RemoveContainer" containerID="bae9b1c35eb766b40134bdf16b3c6574103c0de57ab6bc87c8a8d78f604896ff"
Oct 30 23:00:21.925160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.925130    4581 scope.go:122] "RemoveContainer" containerID="9ad9225af9700caaf2c5e0020e266a5c207543812b4716c2d83a62ab4147498b"
Oct 30 23:00:21.944343 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.943964    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:21.974296 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.974168    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"]
Oct 30 23:00:21.984210 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:21.984168    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-54114e-29890020-crxhb"]
Oct 30 23:00:22.003693 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.003626    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"]
Oct 30 23:00:22.016949 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.016911    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"]
Oct 30 23:00:22.023165 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.023137    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29890020-58qmb"]
Oct 30 23:00:22.025139 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.025112    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d2e83a-4650-437c-a8fb-422c4931413f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d2e83a-4650-437c-a8fb-422c4931413f-maas-api-service-ca\") pod \"36d2e83a-4650-437c-a8fb-422c4931413f\" (UID: \"36d2e83a-4650-437c-a8fb-422c4931413f\") "
Oct 30 23:00:22.025419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.025398    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d2e83a-4650-437c-a8fb-422c4931413f-kube-api-access-lnq4q\" (UniqueName: \"kubernetes.io/projected/36d2e83a-4650-437c-a8fb-422c4931413f-kube-api-access-lnq4q\") pod \"36d2e83a-4650-437c-a8fb-422c4931413f\" (UID: \"36d2e83a-4650-437c-a8fb-422c4931413f\") "
Oct 30 23:00:22.025991 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.025643    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d2e83a-4650-437c-a8fb-422c4931413f-maas-api-service-ca" pod "36d2e83a-4650-437c-a8fb-422c4931413f" (UID: "36d2e83a-4650-437c-a8fb-422c4931413f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:22.026271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.026201    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d2e83a-4650-437c-a8fb-422c4931413f-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.030494 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.030468    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d2e83a-4650-437c-a8fb-422c4931413f-kube-api-access-lnq4q" pod "36d2e83a-4650-437c-a8fb-422c4931413f" (UID: "36d2e83a-4650-437c-a8fb-422c4931413f"). InnerVolumeSpecName "kube-api-access-lnq4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:22.074102 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.074012    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:22.109929 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.109894    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" path="/var/lib/kubelet/pods/26267bf8-2af8-45fc-8640-6a0a3812ee2f/volumes"
Oct 30 23:00:22.110594 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.110561    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" path="/var/lib/kubelet/pods/314f0b61-51e0-463d-8ee3-5e2cfc934690/volumes"
Oct 30 23:00:22.111128 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.111108    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" path="/var/lib/kubelet/pods/4663c26d-c795-4d52-9fdd-47b91bd76fe1/volumes"
Oct 30 23:00:22.112725 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.112105    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" path="/var/lib/kubelet/pods/48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9/volumes"
Oct 30 23:00:22.112725 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.112717    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" path="/var/lib/kubelet/pods/db78128e-6cd0-445d-a667-075d7024ac7a/volumes"
Oct 30 23:00:22.113286 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.113262    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" path="/var/lib/kubelet/pods/e2671f5c-fcc0-46f8-8059-a8da58102f7b/volumes"
Oct 30 23:00:22.120189 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.120162    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"]
Oct 30 23:00:22.132287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.130802    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70232370-49f1-43ba-a855-c3ed07d277b8-kube-api-access-mcwjt\" (UniqueName: \"kubernetes.io/projected/70232370-49f1-43ba-a855-c3ed07d277b8-kube-api-access-mcwjt\") pod \"70232370-49f1-43ba-a855-c3ed07d277b8\" (UID: \"70232370-49f1-43ba-a855-c3ed07d277b8\") "
Oct 30 23:00:22.132287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.130956    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70232370-49f1-43ba-a855-c3ed07d277b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70232370-49f1-43ba-a855-c3ed07d277b8-maas-api-service-ca\") pod \"70232370-49f1-43ba-a855-c3ed07d277b8\" (UID: \"70232370-49f1-43ba-a855-c3ed07d277b8\") "
Oct 30 23:00:22.132287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.132175    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnq4q\" (UniqueName: \"kubernetes.io/projected/36d2e83a-4650-437c-a8fb-422c4931413f-kube-api-access-lnq4q\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.132614 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.132586    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70232370-49f1-43ba-a855-c3ed07d277b8-maas-api-service-ca" pod "70232370-49f1-43ba-a855-c3ed07d277b8" (UID: "70232370-49f1-43ba-a855-c3ed07d277b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:22.136838 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.136808    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70232370-49f1-43ba-a855-c3ed07d277b8-kube-api-access-mcwjt" pod "70232370-49f1-43ba-a855-c3ed07d277b8" (UID: "70232370-49f1-43ba-a855-c3ed07d277b8"). InnerVolumeSpecName "kube-api-access-mcwjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:22.151461 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.151373    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:22.201110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.201081    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.232826    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c80734-09bd-439c-baf5-95906d4a7bf0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c80734-09bd-439c-baf5-95906d4a7bf0-maas-api-service-ca\") pod \"d7c80734-09bd-439c-baf5-95906d4a7bf0\" (UID: \"d7c80734-09bd-439c-baf5-95906d4a7bf0\") "
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.232889    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564d8a9f-e41c-41f5-8a03-90277087de3f-kube-api-access-x9vx2\" (UniqueName: \"kubernetes.io/projected/564d8a9f-e41c-41f5-8a03-90277087de3f-kube-api-access-x9vx2\") pod \"564d8a9f-e41c-41f5-8a03-90277087de3f\" (UID: \"564d8a9f-e41c-41f5-8a03-90277087de3f\") "
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.232991    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c80734-09bd-439c-baf5-95906d4a7bf0-kube-api-access-42zc7\" (UniqueName: \"kubernetes.io/projected/d7c80734-09bd-439c-baf5-95906d4a7bf0-kube-api-access-42zc7\") pod \"d7c80734-09bd-439c-baf5-95906d4a7bf0\" (UID: \"d7c80734-09bd-439c-baf5-95906d4a7bf0\") "
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.233178    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564d8a9f-e41c-41f5-8a03-90277087de3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/564d8a9f-e41c-41f5-8a03-90277087de3f-maas-api-service-ca\") pod \"564d8a9f-e41c-41f5-8a03-90277087de3f\" (UID: \"564d8a9f-e41c-41f5-8a03-90277087de3f\") "
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.233247    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c80734-09bd-439c-baf5-95906d4a7bf0-maas-api-service-ca" pod "d7c80734-09bd-439c-baf5-95906d4a7bf0" (UID: "d7c80734-09bd-439c-baf5-95906d4a7bf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.233837    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70232370-49f1-43ba-a855-c3ed07d277b8-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.233857    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwjt\" (UniqueName: \"kubernetes.io/projected/70232370-49f1-43ba-a855-c3ed07d277b8-kube-api-access-mcwjt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.233872    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c80734-09bd-439c-baf5-95906d4a7bf0-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.238289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.234641    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564d8a9f-e41c-41f5-8a03-90277087de3f-maas-api-service-ca" pod "564d8a9f-e41c-41f5-8a03-90277087de3f" (UID: "564d8a9f-e41c-41f5-8a03-90277087de3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:22.245421 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.245398    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c80734-09bd-439c-baf5-95906d4a7bf0-kube-api-access-42zc7" pod "d7c80734-09bd-439c-baf5-95906d4a7bf0" (UID: "d7c80734-09bd-439c-baf5-95906d4a7bf0"). InnerVolumeSpecName "kube-api-access-42zc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:22.247584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.247555    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564d8a9f-e41c-41f5-8a03-90277087de3f-kube-api-access-x9vx2" pod "564d8a9f-e41c-41f5-8a03-90277087de3f" (UID: "564d8a9f-e41c-41f5-8a03-90277087de3f"). InnerVolumeSpecName "kube-api-access-x9vx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:22.268775 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.268736    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"]
Oct 30 23:00:22.319089 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.319051    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:22.335675 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.335640    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9vx2\" (UniqueName: \"kubernetes.io/projected/564d8a9f-e41c-41f5-8a03-90277087de3f-kube-api-access-x9vx2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.335860 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.335679    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42zc7\" (UniqueName: \"kubernetes.io/projected/d7c80734-09bd-439c-baf5-95906d4a7bf0-kube-api-access-42zc7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.335860 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.335696    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/564d8a9f-e41c-41f5-8a03-90277087de3f-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.387311 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.387279    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:22.436864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.436827    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339358cd-ae97-49d2-a336-13be6d694007-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339358cd-ae97-49d2-a336-13be6d694007-maas-api-service-ca\") pod \"339358cd-ae97-49d2-a336-13be6d694007\" (UID: \"339358cd-ae97-49d2-a336-13be6d694007\") "
Oct 30 23:00:22.437037 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.436877    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2d57df-edc4-4ffa-9308-4432b482085c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2d57df-edc4-4ffa-9308-4432b482085c-maas-api-service-ca\") pod \"2a2d57df-edc4-4ffa-9308-4432b482085c\" (UID: \"2a2d57df-edc4-4ffa-9308-4432b482085c\") "
Oct 30 23:00:22.437087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.437070    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339358cd-ae97-49d2-a336-13be6d694007-kube-api-access-gj9ct\" (UniqueName: \"kubernetes.io/projected/339358cd-ae97-49d2-a336-13be6d694007-kube-api-access-gj9ct\") pod \"339358cd-ae97-49d2-a336-13be6d694007\" (UID: \"339358cd-ae97-49d2-a336-13be6d694007\") "
Oct 30 23:00:22.437160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.437139    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2d57df-edc4-4ffa-9308-4432b482085c-kube-api-access-m9p7j\" (UniqueName: \"kubernetes.io/projected/2a2d57df-edc4-4ffa-9308-4432b482085c-kube-api-access-m9p7j\") pod \"2a2d57df-edc4-4ffa-9308-4432b482085c\" (UID: \"2a2d57df-edc4-4ffa-9308-4432b482085c\") "
Oct 30 23:00:22.437428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.437202    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339358cd-ae97-49d2-a336-13be6d694007-maas-api-service-ca" pod "339358cd-ae97-49d2-a336-13be6d694007" (UID: "339358cd-ae97-49d2-a336-13be6d694007"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:22.437541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.437518    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2d57df-edc4-4ffa-9308-4432b482085c-maas-api-service-ca" pod "2a2d57df-edc4-4ffa-9308-4432b482085c" (UID: "2a2d57df-edc4-4ffa-9308-4432b482085c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:22.438496 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.438467    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339358cd-ae97-49d2-a336-13be6d694007-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.438605 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.438498    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2d57df-edc4-4ffa-9308-4432b482085c-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.439694 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.439664    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2d57df-edc4-4ffa-9308-4432b482085c-kube-api-access-m9p7j" pod "2a2d57df-edc4-4ffa-9308-4432b482085c" (UID: "2a2d57df-edc4-4ffa-9308-4432b482085c"). InnerVolumeSpecName "kube-api-access-m9p7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:22.439872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.439847    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339358cd-ae97-49d2-a336-13be6d694007-kube-api-access-gj9ct" pod "339358cd-ae97-49d2-a336-13be6d694007" (UID: "339358cd-ae97-49d2-a336-13be6d694007"). InnerVolumeSpecName "kube-api-access-gj9ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:22.540055 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.539966    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9ct\" (UniqueName: \"kubernetes.io/projected/339358cd-ae97-49d2-a336-13be6d694007-kube-api-access-gj9ct\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.540055 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.540000    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9p7j\" (UniqueName: \"kubernetes.io/projected/2a2d57df-edc4-4ffa-9308-4432b482085c-kube-api-access-m9p7j\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:22.754045 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.754017    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"
Oct 30 23:00:22.754657 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.754031    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7" event={"ID":"339358cd-ae97-49d2-a336-13be6d694007","Type":"ContainerDied","Data":"e7adb13b8badae8639d24a2dafc366b7ac4fbe8ec195b8c568275257c2a58a63"}
Oct 30 23:00:22.754887 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.754872    4581 scope.go:122] "RemoveContainer" containerID="e7ba340546cd2e94820410f86f774a9b3da60398e95c8527f6ea8108955718b9"
Oct 30 23:00:22.757542 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.757475    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs" event={"ID":"564d8a9f-e41c-41f5-8a03-90277087de3f","Type":"ContainerDied","Data":"77d07478ef6f8a7a082aad6d1b188996084592899611fbefaa1960d29d6690c3"}
Oct 30 23:00:22.757670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.757601    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"
Oct 30 23:00:22.758088 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.758065    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"]
Oct 30 23:00:22.760433 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.760409    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd" event={"ID":"d7c80734-09bd-439c-baf5-95906d4a7bf0","Type":"ContainerDied","Data":"c1fb0eac8379045618bb96d6b6fd1009f595a164d45f9c1550d845a4743195f2"}
Oct 30 23:00:22.760539 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.760518    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"
Oct 30 23:00:22.764282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.763967    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"
Oct 30 23:00:22.764282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.764006    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw" event={"ID":"70232370-49f1-43ba-a855-c3ed07d277b8","Type":"ContainerDied","Data":"3a76e06ceb260b1117b7cbdbccf82dc5aceec007c36bdcf53b6f71bbe8f58c8f"}
Oct 30 23:00:22.770522 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.770500    4581 scope.go:122] "RemoveContainer" containerID="b15ce24b117c2368f6e6a94c1649a914ac153840a589e1186ce79fb40cbe6536"
Oct 30 23:00:22.779341 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.779304    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz" event={"ID":"2a2d57df-edc4-4ffa-9308-4432b482085c","Type":"ContainerDied","Data":"57bc91c26f6f4d2d1dd9c4e9f58dad3e43e18cbc14f1285af565421dd2f4edde"}
Oct 30 23:00:22.779462 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.779348    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"
Oct 30 23:00:22.791953 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.791925    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h" event={"ID":"36d2e83a-4650-437c-a8fb-422c4931413f","Type":"ContainerDied","Data":"e4a9f10e0ae65913c9f471a956af3df504442b44d0493f6fd5b4c7f3025117d5"}
Oct 30 23:00:22.792073 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.792040    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"
Oct 30 23:00:22.792132 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.792074    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"]
Oct 30 23:00:22.792815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.792797    4581 scope.go:122] "RemoveContainer" containerID="eede7678d8810d769e06e7d30e34908ae15b706d9f6c0a884fb0e5f4b01b239a"
Oct 30 23:00:22.808190 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.808164    4581 scope.go:122] "RemoveContainer" containerID="4e840a10f25c6bb280f5f5ccc16c1b0c6ec3d4878ba258b7ddabf9f1f92dbf4d"
Oct 30 23:00:22.821150 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.821125    4581 scope.go:122] "RemoveContainer" containerID="40133ec74bde6cf70e26846d51f3ce693269b23f5d89d4dc463d40f1b64c5ae1"
Oct 30 23:00:22.834760 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.834727    4581 scope.go:122] "RemoveContainer" containerID="07b31218336e73bff624ae35cce715a274bed774f1b738de987baf1ff3bd42e1"
Oct 30 23:00:22.843123 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.843064    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"]
Oct 30 23:00:22.849364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.849335    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5983ea-29890020-xv9hz"]
Oct 30 23:00:22.918279 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.918244    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"]
Oct 30 23:00:22.928108 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.928080    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d9f1c2-29890020-7ngf7"]
Oct 30 23:00:22.985467 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.985438    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:22.996244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:22.992496    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"]
Oct 30 23:00:23.003168 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.003131    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-510f80-29890020-n4d6h"]
Oct 30 23:00:23.024940 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.024912    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:23.030468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.030407    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"]
Oct 30 23:00:23.036183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.036158    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bed544-29890020-cv2hw"]
Oct 30 23:00:23.046737 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.046672    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1797edd-fa79-435c-9e29-3b93095377e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1797edd-fa79-435c-9e29-3b93095377e1-maas-api-service-ca\") pod \"d1797edd-fa79-435c-9e29-3b93095377e1\" (UID: \"d1797edd-fa79-435c-9e29-3b93095377e1\") "
Oct 30 23:00:23.046895 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.046881    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1797edd-fa79-435c-9e29-3b93095377e1-kube-api-access-l4qnz\" (UniqueName: \"kubernetes.io/projected/d1797edd-fa79-435c-9e29-3b93095377e1-kube-api-access-l4qnz\") pod \"d1797edd-fa79-435c-9e29-3b93095377e1\" (UID: \"d1797edd-fa79-435c-9e29-3b93095377e1\") "
Oct 30 23:00:23.047060 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.047034    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1797edd-fa79-435c-9e29-3b93095377e1-maas-api-service-ca" pod "d1797edd-fa79-435c-9e29-3b93095377e1" (UID: "d1797edd-fa79-435c-9e29-3b93095377e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:23.047413 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.047395    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1797edd-fa79-435c-9e29-3b93095377e1-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:23.049311 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.049288    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1797edd-fa79-435c-9e29-3b93095377e1-kube-api-access-l4qnz" pod "d1797edd-fa79-435c-9e29-3b93095377e1" (UID: "d1797edd-fa79-435c-9e29-3b93095377e1"). InnerVolumeSpecName "kube-api-access-l4qnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:23.055936 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.055912    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"]
Oct 30 23:00:23.065234 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.065192    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"]
Oct 30 23:00:23.072347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.072311    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6af86e-29890020-tsrrs"]
Oct 30 23:00:23.099292 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.099255    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"]
Oct 30 23:00:23.105057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.105030    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-0ff5b0-29890020-92qgd"]
Oct 30 23:00:23.152182 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.152147    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8e28dd-5a24-420b-8233-38b38e2bfda3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8e28dd-5a24-420b-8233-38b38e2bfda3-maas-api-service-ca\") pod \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\" (UID: \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\") "
Oct 30 23:00:23.152443 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.152417    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8e28dd-5a24-420b-8233-38b38e2bfda3-kube-api-access-m6nwh\" (UniqueName: \"kubernetes.io/projected/bb8e28dd-5a24-420b-8233-38b38e2bfda3-kube-api-access-m6nwh\") pod \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\" (UID: \"bb8e28dd-5a24-420b-8233-38b38e2bfda3\") "
Oct 30 23:00:23.153276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.153200    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8e28dd-5a24-420b-8233-38b38e2bfda3-maas-api-service-ca" pod "bb8e28dd-5a24-420b-8233-38b38e2bfda3" (UID: "bb8e28dd-5a24-420b-8233-38b38e2bfda3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:23.154907 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.154368    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4qnz\" (UniqueName: \"kubernetes.io/projected/d1797edd-fa79-435c-9e29-3b93095377e1-kube-api-access-l4qnz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:23.154907 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.154397    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8e28dd-5a24-420b-8233-38b38e2bfda3-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:23.158079 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.158052    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8e28dd-5a24-420b-8233-38b38e2bfda3-kube-api-access-m6nwh" pod "bb8e28dd-5a24-420b-8233-38b38e2bfda3" (UID: "bb8e28dd-5a24-420b-8233-38b38e2bfda3"). InnerVolumeSpecName "kube-api-access-m6nwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:23.271672 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.271637    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6nwh\" (UniqueName: \"kubernetes.io/projected/bb8e28dd-5a24-420b-8233-38b38e2bfda3-kube-api-access-m6nwh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:23.282204 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.282180    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:23.372337 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.372306    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-maas-api-service-ca\") pod \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\" (UID: \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\") "
Oct 30 23:00:23.372661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.372639    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-kube-api-access-rlmvv\" (UniqueName: \"kubernetes.io/projected/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-kube-api-access-rlmvv\") pod \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\" (UID: \"aaa9064d-b09a-4b48-b007-0cf8e3c1efac\") "
Oct 30 23:00:23.373632 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.372699    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-maas-api-service-ca" pod "aaa9064d-b09a-4b48-b007-0cf8e3c1efac" (UID: "aaa9064d-b09a-4b48-b007-0cf8e3c1efac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:23.376346 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.376320    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-kube-api-access-rlmvv" pod "aaa9064d-b09a-4b48-b007-0cf8e3c1efac" (UID: "aaa9064d-b09a-4b48-b007-0cf8e3c1efac"). InnerVolumeSpecName "kube-api-access-rlmvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:23.474596 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.474564    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:23.474596 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.474594    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlmvv\" (UniqueName: \"kubernetes.io/projected/aaa9064d-b09a-4b48-b007-0cf8e3c1efac-kube-api-access-rlmvv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:23.806382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.806285    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"
Oct 30 23:00:23.806382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.806294    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4" event={"ID":"aaa9064d-b09a-4b48-b007-0cf8e3c1efac","Type":"ContainerDied","Data":"e54ee8f55168a245343c3704efb85b7ec0654f9f16608f16d318075a9adeb9eb"}
Oct 30 23:00:23.806921 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.806388    4581 scope.go:122] "RemoveContainer" containerID="8d414bb66524bfb90aa3459903d174eeb393fe8ac309a4d7584bda9a1f4f41e0"
Oct 30 23:00:23.809168 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.809147    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"
Oct 30 23:00:23.809313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.809162    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4" event={"ID":"bb8e28dd-5a24-420b-8233-38b38e2bfda3","Type":"ContainerDied","Data":"0cfd5418bf541005469749bec479c29a7d6fa9e8f5426bf23a1591b4992c5563"}
Oct 30 23:00:23.811680 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.811660    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"
Oct 30 23:00:23.811680 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.811665    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w" event={"ID":"d1797edd-fa79-435c-9e29-3b93095377e1","Type":"ContainerDied","Data":"8067b430ce826088e93d1ae6201a231da40528813c8baf6a54068071b5169c51"}
Oct 30 23:00:23.823630 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.823586    4581 scope.go:122] "RemoveContainer" containerID="62082a5fdb911ffcc2c0e9b6651e3ff1f9aba71b183c59a22f76ea728ad45eb7"
Oct 30 23:00:23.835626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.835610    4581 scope.go:122] "RemoveContainer" containerID="2590fe7d761d5b7e4f07a41542128cdd231d815f55b3d4f970833b64b12b73e4"
Oct 30 23:00:23.878577 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.878550    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"]
Oct 30 23:00:23.884571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.884543    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-421583-29890020-v8jf4"]
Oct 30 23:00:23.912936 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.912906    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"]
Oct 30 23:00:23.932058 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.932022    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cdc122-29890020-rl82w"]
Oct 30 23:00:23.963474 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.963441    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"]
Oct 30 23:00:23.977612 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:23.977580    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e14e00-29890020-9pds4"]
Oct 30 23:00:24.099476 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.099400    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" path="/var/lib/kubelet/pods/2a2d57df-edc4-4ffa-9308-4432b482085c/volumes"
Oct 30 23:00:24.099851 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.099836    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339358cd-ae97-49d2-a336-13be6d694007" path="/var/lib/kubelet/pods/339358cd-ae97-49d2-a336-13be6d694007/volumes"
Oct 30 23:00:24.100244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.100229    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" path="/var/lib/kubelet/pods/36d2e83a-4650-437c-a8fb-422c4931413f/volumes"
Oct 30 23:00:24.100945 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.100929    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" path="/var/lib/kubelet/pods/564d8a9f-e41c-41f5-8a03-90277087de3f/volumes"
Oct 30 23:00:24.101375 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.101359    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" path="/var/lib/kubelet/pods/70232370-49f1-43ba-a855-c3ed07d277b8/volumes"
Oct 30 23:00:24.101843 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.101826    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" path="/var/lib/kubelet/pods/aaa9064d-b09a-4b48-b007-0cf8e3c1efac/volumes"
Oct 30 23:00:24.102498 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.102483    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" path="/var/lib/kubelet/pods/bb8e28dd-5a24-420b-8233-38b38e2bfda3/volumes"
Oct 30 23:00:24.102868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.102853    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" path="/var/lib/kubelet/pods/d1797edd-fa79-435c-9e29-3b93095377e1/volumes"
Oct 30 23:00:24.103308 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:24.103290    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" path="/var/lib/kubelet/pods/d7c80734-09bd-439c-baf5-95906d4a7bf0/volumes"
Oct 30 23:00:29.031557 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:29.031519    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"]
Oct 30 23:00:29.041015 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:29.040986    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976d46b7-lnxn5"]
Oct 30 23:00:30.113279 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:30.113198    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efde7b5e-b65f-450f-b47c-7e5fb76f8e1d" path="/var/lib/kubelet/pods/efde7b5e-b65f-450f-b47c-7e5fb76f8e1d/volumes"
Oct 30 23:00:31.031960 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:31.031925    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"]
Oct 30 23:00:31.041823 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:31.041788    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7a15-tcpbd"]
Oct 30 23:00:31.047487 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:31.047455    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"]
Oct 30 23:00:31.053365 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:31.053335    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae9e4fe5-4hrjq"]
Oct 30 23:00:32.100393 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:32.100358    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4" path="/var/lib/kubelet/pods/bdc1a3ed-ff96-4d3d-86ba-4da37b7c3fd4/volumes"
Oct 30 23:00:32.100880 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:32.100865    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed08f5ea-cca4-4188-b81d-e2b2f8f5003b" path="/var/lib/kubelet/pods/ed08f5ea-cca4-4188-b81d-e2b2f8f5003b/volumes"
Oct 30 23:00:34.683365 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:34.683325    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"]
Oct 30 23:00:35.038381 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:35.038295    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"]
Oct 30 23:00:35.044878 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:35.044848    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0728251b-2jc45"]
Oct 30 23:00:36.106530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:36.106489    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d3d612-28fb-49c6-bc10-d6b305f688f1" path="/var/lib/kubelet/pods/07d3d612-28fb-49c6-bc10-d6b305f688f1/volumes"
Oct 30 23:00:37.302059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.302018    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78" podUID="f8e2b178-91ce-4aa8-bf90-64b6066054f5" containerName="main" containerID="cri-o://607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50" gracePeriod=45
Oct 30 23:00:37.590639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.590610    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 23:00:37.637333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.637304    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e2b178-91ce-4aa8-bf90-64b6066054f5-kube-api-access-4sdgd\" (UniqueName: \"kubernetes.io/projected/f8e2b178-91ce-4aa8-bf90-64b6066054f5-kube-api-access-4sdgd\") pod \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") "
Oct 30 23:00:37.637494 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.637411    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-model-cache\") pod \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") "
Oct 30 23:00:37.637494 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.637473    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tls-certs\") pod \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") "
Oct 30 23:00:37.637606 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.637556    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tmp-dir\") pod \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") "
Oct 30 23:00:37.637758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.637725    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-model-cache" pod "f8e2b178-91ce-4aa8-bf90-64b6066054f5" (UID: "f8e2b178-91ce-4aa8-bf90-64b6066054f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:00:37.637870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.637745    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-dshm\") pod \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") "
Oct 30 23:00:37.637870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.637848    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tmp-dir" pod "f8e2b178-91ce-4aa8-bf90-64b6066054f5" (UID: "f8e2b178-91ce-4aa8-bf90-64b6066054f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:00:37.637870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.637861    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-home\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-home\") pod \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\" (UID: \"f8e2b178-91ce-4aa8-bf90-64b6066054f5\") "
Oct 30 23:00:37.638196 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.638159    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-home" pod "f8e2b178-91ce-4aa8-bf90-64b6066054f5" (UID: "f8e2b178-91ce-4aa8-bf90-64b6066054f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:00:37.638622 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.638592    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:37.638683 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.638620    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:37.638683 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.638637    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:37.641257 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.641192    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e2b178-91ce-4aa8-bf90-64b6066054f5-kube-api-access-4sdgd" pod "f8e2b178-91ce-4aa8-bf90-64b6066054f5" (UID: "f8e2b178-91ce-4aa8-bf90-64b6066054f5"). InnerVolumeSpecName "kube-api-access-4sdgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:37.641491 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.641458    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-dshm" pod "f8e2b178-91ce-4aa8-bf90-64b6066054f5" (UID: "f8e2b178-91ce-4aa8-bf90-64b6066054f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:00:37.641491 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.641467    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tls-certs" pod "f8e2b178-91ce-4aa8-bf90-64b6066054f5" (UID: "f8e2b178-91ce-4aa8-bf90-64b6066054f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:00:37.739877 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.739848    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e2b178-91ce-4aa8-bf90-64b6066054f5-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:37.739877 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.739872    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e2b178-91ce-4aa8-bf90-64b6066054f5-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:37.739877 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.739881    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sdgd\" (UniqueName: \"kubernetes.io/projected/f8e2b178-91ce-4aa8-bf90-64b6066054f5-kube-api-access-4sdgd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:37.962609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.962571    4581 generic.go:356] "Generic (PLEG): container finished" podID="f8e2b178-91ce-4aa8-bf90-64b6066054f5" containerID="607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50" exitCode=0
Oct 30 23:00:37.962842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.962631    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"
Oct 30 23:00:37.962842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.962664    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78" event={"ID":"f8e2b178-91ce-4aa8-bf90-64b6066054f5","Type":"ContainerDied","Data":"607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50"}
Oct 30 23:00:37.962842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.962749    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78" event={"ID":"f8e2b178-91ce-4aa8-bf90-64b6066054f5","Type":"ContainerDied","Data":"c52a8cb5969836fa1aad80a02eee145d13161ac23e5f43e6a497258391b4861b"}
Oct 30 23:00:37.962842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.962774    4581 scope.go:122] "RemoveContainer" containerID="607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50"
Oct 30 23:00:37.977855 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.977838    4581 scope.go:122] "RemoveContainer" containerID="607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50"
Oct 30 23:00:37.978163 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:37.978142    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50\": container with ID starting with 607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50 not found: ID does not exist" containerID="607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50"
Oct 30 23:00:37.978274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:37.978171    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50"} err="failed to get container status \"607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50\": rpc error: code = NotFound desc = could not find container \"607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50\": container with ID starting with 607c6b5ab9cecac33878c3b4d8c16d167179a1271cbccc3e130f2e46ecf68e50 not found: ID does not exist"
Oct 30 23:00:38.028948 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:38.028912    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"]
Oct 30 23:00:38.033615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:38.033593    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e3a7dd/ipp-route-e3a7dd-kserve-788f9dbc89-5fg78"]
Oct 30 23:00:38.101476 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:38.101434    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e2b178-91ce-4aa8-bf90-64b6066054f5" path="/var/lib/kubelet/pods/f8e2b178-91ce-4aa8-bf90-64b6066054f5/volumes"
Oct 30 23:00:39.032197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.032163    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"]
Oct 30 23:00:39.041460 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.041432    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ad67ad-nw4lq"]
Oct 30 23:00:39.684318 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.684259    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn" podUID="ea6fed26-0cef-4d4a-9d3e-76643a5215dd" containerName="payload-pre-processing" containerID="cri-o://460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453" gracePeriod=25
Oct 30 23:00:39.970802 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.970774    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 23:00:39.986875 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.986841    4581 generic.go:356] "Generic (PLEG): container finished" podID="ea6fed26-0cef-4d4a-9d3e-76643a5215dd" containerID="460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453" exitCode=0
Oct 30 23:00:39.987086 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.986889    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"
Oct 30 23:00:39.987086 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.986928    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn" event={"ID":"ea6fed26-0cef-4d4a-9d3e-76643a5215dd","Type":"ContainerDied","Data":"460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453"}
Oct 30 23:00:39.987086 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.987023    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn" event={"ID":"ea6fed26-0cef-4d4a-9d3e-76643a5215dd","Type":"ContainerDied","Data":"9ccdc8f571cd0a53e34be1d6119f737b0c25fc5f2caefb4b030ef551092ee941"}
Oct 30 23:00:39.987086 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.987054    4581 scope.go:122] "RemoveContainer" containerID="460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453"
Oct 30 23:00:39.998997 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.998973    4581 scope.go:122] "RemoveContainer" containerID="460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453"
Oct 30 23:00:39.999379 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:39.999356    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453\": container with ID starting with 460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453 not found: ID does not exist" containerID="460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453"
Oct 30 23:00:39.999483 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:39.999386    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453"} err="failed to get container status \"460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453\": rpc error: code = NotFound desc = could not find container \"460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453\": container with ID starting with 460cee076f34724104c63020dd1f82028362ca092eac2c71d97065ccda10b453 not found: ID does not exist"
Oct 30 23:00:40.080882 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.080851    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-kube-api-access-vd8zf\" (UniqueName: \"kubernetes.io/projected/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-kube-api-access-vd8zf\") pod \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\" (UID: \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\") "
Oct 30 23:00:40.081377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.081024    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-plugins-config-volume\") pod \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\" (UID: \"ea6fed26-0cef-4d4a-9d3e-76643a5215dd\") "
Oct 30 23:00:40.081505 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.081480    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-plugins-config-volume" pod "ea6fed26-0cef-4d4a-9d3e-76643a5215dd" (UID: "ea6fed26-0cef-4d4a-9d3e-76643a5215dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:40.081658 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.081641    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:40.083526 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.083497    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-kube-api-access-vd8zf" pod "ea6fed26-0cef-4d4a-9d3e-76643a5215dd" (UID: "ea6fed26-0cef-4d4a-9d3e-76643a5215dd"). InnerVolumeSpecName "kube-api-access-vd8zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:40.101118 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.101091    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f1aba2-1a40-496d-bf4b-1a6134294f8c" path="/var/lib/kubelet/pods/54f1aba2-1a40-496d-bf4b-1a6134294f8c/volumes"
Oct 30 23:00:40.183593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.183555    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8zf\" (UniqueName: \"kubernetes.io/projected/ea6fed26-0cef-4d4a-9d3e-76643a5215dd-kube-api-access-vd8zf\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:40.326415 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.326329    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"]
Oct 30 23:00:40.331340 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:40.331312    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-5c684bc9-8ccfn"]
Oct 30 23:00:40.818194 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:00:40.818144    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:00:41.054845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:41.054811    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"]
Oct 30 23:00:41.066785 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:41.066752    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"]
Oct 30 23:00:41.074627 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:41.074550    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745130f-rg6qs"]
Oct 30 23:00:41.082020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:41.081990    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330db704-tj4w2"]
Oct 30 23:00:42.101751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:42.101718    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5c3a1a-a078-4076-bad2-7bbac28de4f8" path="/var/lib/kubelet/pods/4e5c3a1a-a078-4076-bad2-7bbac28de4f8/volumes"
Oct 30 23:00:42.102284 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:42.102269    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7574ffd0-8f3a-4720-84c6-98a8dbaf7813" path="/var/lib/kubelet/pods/7574ffd0-8f3a-4720-84c6-98a8dbaf7813/volumes"
Oct 30 23:00:42.102661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:42.102648    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6fed26-0cef-4d4a-9d3e-76643a5215dd" path="/var/lib/kubelet/pods/ea6fed26-0cef-4d4a-9d3e-76643a5215dd/volumes"
Oct 30 23:00:46.029812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:46.029770    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"]
Oct 30 23:00:46.039619 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:46.039575    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24295132-qbh7m"]
Oct 30 23:00:46.099857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:46.099810    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70aae6ab-0b1c-45e4-b7f9-38e86bcf142b" path="/var/lib/kubelet/pods/70aae6ab-0b1c-45e4-b7f9-38e86bcf142b/volumes"
Oct 30 23:00:51.032621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.032582    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"]
Oct 30 23:00:51.042762 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.042619    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88d9b76-vpwkr"]
Oct 30 23:00:51.775644 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.775556    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"]
Oct 30 23:00:51.776331 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776314    4581 cpu_manager.go:414] "containerMap: removing container" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:00:51.776331 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776331    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776342    4581 cpu_manager.go:414] "containerMap: removing container" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776348    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776356    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ea6fed26-0cef-4d4a-9d3e-76643a5215dd" containerName="payload-pre-processing"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776361    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6fed26-0cef-4d4a-9d3e-76643a5215dd" containerName="payload-pre-processing"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776376    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776380    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776389    4581 cpu_manager.go:414] "containerMap: removing container" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776394    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776400    4581 cpu_manager.go:414] "containerMap: removing container" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776405    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776417    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776421    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776427    4581 cpu_manager.go:414] "containerMap: removing container" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776432    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776438    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776442    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776449    4581 cpu_manager.go:414] "containerMap: removing container" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776453    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776460    4581 cpu_manager.go:414] "containerMap: removing container" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776463    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776471    4581 cpu_manager.go:414] "containerMap: removing container" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:00:51.776468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776475    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776482    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776486    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776494    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776498    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776503    4581 cpu_manager.go:414] "containerMap: removing container" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776507    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776517    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776521    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776528    4581 cpu_manager.go:414] "containerMap: removing container" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776532    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776539    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776543    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776553    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776557    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776566    4581 cpu_manager.go:414] "containerMap: removing container" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776571    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776585    4581 cpu_manager.go:414] "containerMap: removing container" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776591    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776599    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776603    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776609    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776613    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776619    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776624    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776630    4581 cpu_manager.go:414] "containerMap: removing container" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776633    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776639    4581 cpu_manager.go:414] "containerMap: removing container" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776643    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776652    4581 cpu_manager.go:414] "containerMap: removing container" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776660    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776670    4581 cpu_manager.go:414] "containerMap: removing container" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776674    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776680    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776685    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776694    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776699    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776704    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f8e2b178-91ce-4aa8-bf90-64b6066054f5" containerName="main"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776708    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e2b178-91ce-4aa8-bf90-64b6066054f5" containerName="main"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776715    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776719    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776727    4581 cpu_manager.go:414] "containerMap: removing container" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776732    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776742    4581 cpu_manager.go:414] "containerMap: removing container" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776749    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776756    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776760    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776766    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776771    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776776    4581 cpu_manager.go:414] "containerMap: removing container" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776780    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776786    4581 cpu_manager.go:414] "containerMap: removing container" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776790    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776798    4581 cpu_manager.go:414] "containerMap: removing container" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776802    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776809    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776814    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776830    4581 cpu_manager.go:414] "containerMap: removing container" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776835    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776841    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776845    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776851    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776855    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776961    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776969    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776977    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6fed26-0cef-4d4a-9d3e-76643a5215dd" containerName="payload-pre-processing"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776984    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776990    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.776996    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777002    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777008    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777013    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777018    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777023    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777028    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777034    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777039    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777044    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777050    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777055    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777061    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777066    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777071    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="339358cd-ae97-49d2-a336-13be6d694007" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777076    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777082    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777086    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777092    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777097    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777103    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777107    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777113    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e6e4c4-0302-41c1-a6b9-a71a30e7e2b9" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777118    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777125    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777130    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e2b178-91ce-4aa8-bf90-64b6066054f5" containerName="main"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777136    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777141    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777147    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777152    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777158    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1797edd-fa79-435c-9e29-3b93095377e1" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777164    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777169    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777174    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777179    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:00:51.777485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.777183    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:00:51.781760 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.781000    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:51.806322 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.806293    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:00:51.806551 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.806410    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:00:51.806882 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.806860    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:00:51.807034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.806950    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:00:51.810996 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.810975    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"]
Oct 30 23:00:51.917611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.917575    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:51.917829 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.917649    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:51.917829 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:51.917754    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9qd\" (UniqueName: \"kubernetes.io/projected/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-kube-api-access-5m9qd\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:52.019346 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.019300    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:52.019573 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.019367    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9qd\" (UniqueName: \"kubernetes.io/projected/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-kube-api-access-5m9qd\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:52.019750 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.019729    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:52.020065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.020039    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:52.022534 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.022502    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:52.037592 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.037505    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9qd\" (UniqueName: \"kubernetes.io/projected/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-kube-api-access-5m9qd\") pod \"maas-api-revoke-sub-d1e62679-459p7\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:52.100235 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.100193    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:52.102801 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.102763    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbb8c63-5389-46c3-8d69-46880b94300e" path="/var/lib/kubelet/pods/9dbb8c63-5389-46c3-8d69-46880b94300e/volumes"
Oct 30 23:00:52.436580 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:52.436553    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"]
Oct 30 23:00:52.439563 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:00:52.439530    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31ad848_8ea6_42eb_9aa8_0a8aa4f706aa.slice/crio-69a6d8fc1561ce6105b5ec1b0bc326ec69a5d838d4c2511cfcf81cc325ce3d2e WatchSource:0}: Error finding container 69a6d8fc1561ce6105b5ec1b0bc326ec69a5d838d4c2511cfcf81cc325ce3d2e: Status 404 returned error can't find the container with id 69a6d8fc1561ce6105b5ec1b0bc326ec69a5d838d4c2511cfcf81cc325ce3d2e
Oct 30 23:00:53.070773 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:53.070673    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"]
Oct 30 23:00:53.176416 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:53.176383    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7" event={"ID":"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa","Type":"ContainerStarted","Data":"bd9097e3cdcb612f48b3865c58651717fb994f63561cd17735b08590e42f8da2"}
Oct 30 23:00:53.176593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:53.176445    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7" event={"ID":"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa","Type":"ContainerStarted","Data":"69a6d8fc1561ce6105b5ec1b0bc326ec69a5d838d4c2511cfcf81cc325ce3d2e"}
Oct 30 23:00:53.216436 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:53.216374    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7" podStartSLOduration=2.216351342 podStartE2EDuration="2.216351342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:53.211870216 +0000 UTC m=+3237.250809102" watchObservedRunningTime="2026-10-30 23:00:53.216351342 +0000 UTC m=+3237.255290229"
Oct 30 23:00:54.190329 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:54.190290    4581 generic.go:356] "Generic (PLEG): container finished" podID="d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa" containerID="bd9097e3cdcb612f48b3865c58651717fb994f63561cd17735b08590e42f8da2" exitCode=0
Oct 30 23:00:54.190870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:54.190378    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7" event={"ID":"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa","Type":"ContainerDied","Data":"bd9097e3cdcb612f48b3865c58651717fb994f63561cd17735b08590e42f8da2"}
Oct 30 23:00:55.437314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.437280    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:55.608104 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.608073    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-kube-api-access-5m9qd\" (UniqueName: \"kubernetes.io/projected/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-kube-api-access-5m9qd\") pod \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") "
Oct 30 23:00:55.613715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.608119    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-xks-ca\") pod \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") "
Oct 30 23:00:55.613715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.608375    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-openshift-ca\") pod \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\" (UID: \"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa\") "
Oct 30 23:00:55.613715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.608665    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-openshift-ca" pod "d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa" (UID: "d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:55.613715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.608857    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:55.613715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.610419    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-xks-ca" pod "d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa" (UID: "d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:00:55.613715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.610425    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-kube-api-access-5m9qd" pod "d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa" (UID: "d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa"). InnerVolumeSpecName "kube-api-access-5m9qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:55.710203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.710168    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9qd\" (UniqueName: \"kubernetes.io/projected/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-kube-api-access-5m9qd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:55.710203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:55.710200    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:56.024421 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.024386    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": read tcp 10.133.0.2:38370->10.133.1.79:15021: read: connection reset by peer" start-of-body=
Oct 30 23:00:56.024646 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.024549    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": read tcp 10.133.0.2:38370->10.133.1.79:15021: read: connection reset by peer"
Oct 30 23:00:56.214240 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.214194    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"
Oct 30 23:00:56.214457 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.214246    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7" event={"ID":"d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa","Type":"ContainerDied","Data":"69a6d8fc1561ce6105b5ec1b0bc326ec69a5d838d4c2511cfcf81cc325ce3d2e"}
Oct 30 23:00:56.214457 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.214305    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69a6d8fc1561ce6105b5ec1b0bc326ec69a5d838d4c2511cfcf81cc325ce3d2e"
Oct 30 23:00:56.216350 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.216324    4581 generic.go:356] "Generic (PLEG): container finished" podID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerID="a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c" exitCode=0
Oct 30 23:00:56.216507 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.216408    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" event={"ID":"c069b4a9-3dd5-4144-a15e-e3df78c5f554","Type":"ContainerDied","Data":"a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c"}
Oct 30 23:00:56.216851 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.216833    4581 scope.go:122] "RemoveContainer" containerID="a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c"
Oct 30 23:00:56.218469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.218432    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:00:56.218599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.218523    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:00:56.218599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:56.218573    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:00:57.228694 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:57.228659    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" event={"ID":"c069b4a9-3dd5-4144-a15e-e3df78c5f554","Type":"ContainerStarted","Data":"e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca"}
Oct 30 23:00:57.727611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:57.727562    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 23:00:57.730276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:57.729276    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body=
Oct 30 23:00:57.730276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:57.729335    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused"
Oct 30 23:00:58.076307 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.072284    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs" podUID="c62f335e-cd92-45d1-96b9-ba5e7099cf8e" containerName="payload-pre-processing" containerID="cri-o://54d240d02b1b282af3ac18d7fa3cf67492a22c41f630cb03a77c40c0236e2626" gracePeriod=25
Oct 30 23:00:58.241689 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.241654    4581 generic.go:356] "Generic (PLEG): container finished" podID="c62f335e-cd92-45d1-96b9-ba5e7099cf8e" containerID="54d240d02b1b282af3ac18d7fa3cf67492a22c41f630cb03a77c40c0236e2626" exitCode=0
Oct 30 23:00:58.242173 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.241732    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs" event={"ID":"c62f335e-cd92-45d1-96b9-ba5e7099cf8e","Type":"ContainerDied","Data":"54d240d02b1b282af3ac18d7fa3cf67492a22c41f630cb03a77c40c0236e2626"}
Oct 30 23:00:58.434883 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.434340    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 23:00:58.474262 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.465708    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-kube-api-access-rn8cr\" (UniqueName: \"kubernetes.io/projected/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-kube-api-access-rn8cr\") pod \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\" (UID: \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\") "
Oct 30 23:00:58.474262 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.465985    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-plugins-config-volume\") pod \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\" (UID: \"c62f335e-cd92-45d1-96b9-ba5e7099cf8e\") "
Oct 30 23:00:58.491313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.489464    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-plugins-config-volume" pod "c62f335e-cd92-45d1-96b9-ba5e7099cf8e" (UID: "c62f335e-cd92-45d1-96b9-ba5e7099cf8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:58.491313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.490431    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:58.497255 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.493033    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-kube-api-access-rn8cr" pod "c62f335e-cd92-45d1-96b9-ba5e7099cf8e" (UID: "c62f335e-cd92-45d1-96b9-ba5e7099cf8e"). InnerVolumeSpecName "kube-api-access-rn8cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:58.591851 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.591809    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8cr\" (UniqueName: \"kubernetes.io/projected/c62f335e-cd92-45d1-96b9-ba5e7099cf8e-kube-api-access-rn8cr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:00:58.723830 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.723741    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body=
Oct 30 23:00:58.723830 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:58.723800    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused"
Oct 30 23:00:59.287376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:59.283937    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs" event={"ID":"c62f335e-cd92-45d1-96b9-ba5e7099cf8e","Type":"ContainerDied","Data":"fb5129163c5f86db4b8663958d002a8fdb2b57a23519899c74b50044232c8978"}
Oct 30 23:00:59.287376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:59.284016    4581 scope.go:122] "RemoveContainer" containerID="54d240d02b1b282af3ac18d7fa3cf67492a22c41f630cb03a77c40c0236e2626"
Oct 30 23:00:59.287376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:59.284200    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"
Oct 30 23:00:59.336317 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:59.336282    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"]
Oct 30 23:00:59.346339 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:59.345958    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-b49f58999-qjqrs"]
Oct 30 23:00:59.727547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:59.726173    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body=
Oct 30 23:00:59.727547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:00:59.726268    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused"
Oct 30 23:01:00.113572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:00.106941    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62f335e-cd92-45d1-96b9-ba5e7099cf8e" path="/var/lib/kubelet/pods/c62f335e-cd92-45d1-96b9-ba5e7099cf8e/volumes"
Oct 30 23:01:00.724202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:00.724165    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body=
Oct 30 23:01:00.724613 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:00.724261    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused"
Oct 30 23:01:01.724317 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:01.724283    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused" start-of-body=
Oct 30 23:01:01.724889 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:01.724846    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.79:15021/healthz/ready\": dial tcp 10.133.1.79:15021: connect: connection refused"
Oct 30 23:01:02.032318 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.032207    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"]
Oct 30 23:01:02.042027 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.041993    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0741c530-zdc5m"]
Oct 30 23:01:02.103055 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.103021    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e" path="/var/lib/kubelet/pods/6b46a98a-c1b7-40e2-88ab-b0e27b59bf6e/volumes"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.150212    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"]
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151359    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151380    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f084d37b-3a9e-41b3-8820-ffa93587d8be" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151402    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151409    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa9064d-b09a-4b48-b007-0cf8e3c1efac" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151428    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151437    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151454    4581 cpu_manager.go:414] "containerMap: removing container" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151461    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151475    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151483    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151513    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151519    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4663c26d-c795-4d52-9fdd-47b91bd76fe1" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151530    4581 cpu_manager.go:414] "containerMap: removing container" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151537    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151570    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c62f335e-cd92-45d1-96b9-ba5e7099cf8e" containerName="payload-pre-processing"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151578    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62f335e-cd92-45d1-96b9-ba5e7099cf8e" containerName="payload-pre-processing"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151589    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151596    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151618    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa" containerName="revoke-keys"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151626    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa" containerName="revoke-keys"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151639    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151647    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151805    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="564d8a9f-e41c-41f5-8a03-90277087de3f" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151822    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2671f5c-fcc0-46f8-8059-a8da58102f7b" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151834    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="70232370-49f1-43ba-a855-c3ed07d277b8" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151842    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c80734-09bd-439c-baf5-95906d4a7bf0" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151856    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="db78128e-6cd0-445d-a667-075d7024ac7a" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151865    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="26267bf8-2af8-45fc-8640-6a0a3812ee2f" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151877    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3259e63-46a1-4572-9e46-b249d5fc24e8" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151888    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d2e83a-4650-437c-a8fb-422c4931413f" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151896    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2d57df-edc4-4ffa-9308-4432b482085c" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151906    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62f335e-cd92-45d1-96b9-ba5e7099cf8e" containerName="payload-pre-processing"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151918    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8e28dd-5a24-420b-8233-38b38e2bfda3" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151930    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="314f0b61-51e0-463d-8ee3-5e2cfc934690" containerName="cleanup"
Oct 30 23:01:02.164246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.151939    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa" containerName="revoke-keys"
Oct 30 23:01:02.192325 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.183446    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.197393 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.197121    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"]
Oct 30 23:01:02.197902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.197878    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:01:02.198188 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.198176    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:01:02.198331 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.198321    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:01:02.210609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.201911    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:01:02.293284 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.293243    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.293697 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.293680    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptwl\" (UniqueName: \"kubernetes.io/projected/bff01d2d-904d-4b97-8f67-f928208e138c-kube-api-access-rptwl\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.293797 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.293786    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.396040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.395993    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.396354 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.396331    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptwl\" (UniqueName: \"kubernetes.io/projected/bff01d2d-904d-4b97-8f67-f928208e138c-kube-api-access-rptwl\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.396409 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.396378    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.397445 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.397416    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.399028 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.399001    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.414266 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.414202    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptwl\" (UniqueName: \"kubernetes.io/projected/bff01d2d-904d-4b97-8f67-f928208e138c-kube-api-access-rptwl\") pod \"maas-api-revoke-sub-63c443f6-9cjnc\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.523188 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.523144    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:02.728542 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.728510    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 23:01:02.729041 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.729022    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 23:01:02.729733 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.729711    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 23:01:02.862371 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:02.862334    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"]
Oct 30 23:01:02.873281 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:02.871653    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff01d2d_904d_4b97_8f67_f928208e138c.slice/crio-2083c5762c1e21fe8e2b54c50f4b5c001058ff87acbc14cece7855252c6d01af WatchSource:0}: Error finding container 2083c5762c1e21fe8e2b54c50f4b5c001058ff87acbc14cece7855252c6d01af: Status 404 returned error can't find the container with id 2083c5762c1e21fe8e2b54c50f4b5c001058ff87acbc14cece7855252c6d01af
Oct 30 23:01:03.344991 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:03.344949    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc" event={"ID":"bff01d2d-904d-4b97-8f67-f928208e138c","Type":"ContainerStarted","Data":"2550894cdda162550285e3e33829e022c24c3816c353c1d0dad88093a143ac63"}
Oct 30 23:01:03.345211 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:03.345015    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc" event={"ID":"bff01d2d-904d-4b97-8f67-f928208e138c","Type":"ContainerStarted","Data":"2083c5762c1e21fe8e2b54c50f4b5c001058ff87acbc14cece7855252c6d01af"}
Oct 30 23:01:03.386149 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:03.386089    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc" podStartSLOduration=1.386073372 podStartE2EDuration="1.386073372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:03.380957904 +0000 UTC m=+3247.419896791" watchObservedRunningTime="2026-10-30 23:01:03.386073372 +0000 UTC m=+3247.425012258"
Oct 30 23:01:04.360302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:04.360258    4581 generic.go:356] "Generic (PLEG): container finished" podID="bff01d2d-904d-4b97-8f67-f928208e138c" containerID="2550894cdda162550285e3e33829e022c24c3816c353c1d0dad88093a143ac63" exitCode=0
Oct 30 23:01:04.360792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:04.360509    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc" event={"ID":"bff01d2d-904d-4b97-8f67-f928208e138c","Type":"ContainerDied","Data":"2550894cdda162550285e3e33829e022c24c3816c353c1d0dad88093a143ac63"}
Oct 30 23:01:05.563181 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.563155    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:05.641584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.641488    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-xks-ca\") pod \"bff01d2d-904d-4b97-8f67-f928208e138c\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") "
Oct 30 23:01:05.641584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.641562    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-openshift-ca\") pod \"bff01d2d-904d-4b97-8f67-f928208e138c\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") "
Oct 30 23:01:05.641837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.641650    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff01d2d-904d-4b97-8f67-f928208e138c-kube-api-access-rptwl\" (UniqueName: \"kubernetes.io/projected/bff01d2d-904d-4b97-8f67-f928208e138c-kube-api-access-rptwl\") pod \"bff01d2d-904d-4b97-8f67-f928208e138c\" (UID: \"bff01d2d-904d-4b97-8f67-f928208e138c\") "
Oct 30 23:01:05.642065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.641998    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-openshift-ca" pod "bff01d2d-904d-4b97-8f67-f928208e138c" (UID: "bff01d2d-904d-4b97-8f67-f928208e138c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:05.645263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.642738    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:05.645263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.644574    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-xks-ca" pod "bff01d2d-904d-4b97-8f67-f928208e138c" (UID: "bff01d2d-904d-4b97-8f67-f928208e138c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:05.649377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.649346    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff01d2d-904d-4b97-8f67-f928208e138c-kube-api-access-rptwl" pod "bff01d2d-904d-4b97-8f67-f928208e138c" (UID: "bff01d2d-904d-4b97-8f67-f928208e138c"). InnerVolumeSpecName "kube-api-access-rptwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:05.744139 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.744099    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff01d2d-904d-4b97-8f67-f928208e138c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:05.744139 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:05.744137    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rptwl\" (UniqueName: \"kubernetes.io/projected/bff01d2d-904d-4b97-8f67-f928208e138c-kube-api-access-rptwl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:06.388032 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:06.387937    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"
Oct 30 23:01:06.388032 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:06.387973    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc" event={"ID":"bff01d2d-904d-4b97-8f67-f928208e138c","Type":"ContainerDied","Data":"2083c5762c1e21fe8e2b54c50f4b5c001058ff87acbc14cece7855252c6d01af"}
Oct 30 23:01:06.388296 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:06.388035    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2083c5762c1e21fe8e2b54c50f4b5c001058ff87acbc14cece7855252c6d01af"
Oct 30 23:01:07.200369 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:07.200334    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"]
Oct 30 23:01:07.200824 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:07.200613    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy" containerID="cri-o://97642a3e44417d17a30b89f573939bb7539f03fc959b7b71b073d10c0115553c" gracePeriod=30
Oct 30 23:01:10.818021 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:01:10.817986    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:01:12.076159 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.076110    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:01:12.076715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.076176    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:01:12.481579 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.481548    4581 generic.go:356] "Generic (PLEG): container finished" podID="2b658db4-7498-4462-850d-ba2669a04dac" containerID="97642a3e44417d17a30b89f573939bb7539f03fc959b7b71b073d10c0115553c" exitCode=0
Oct 30 23:01:12.481769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.481624    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" event={"ID":"2b658db4-7498-4462-850d-ba2669a04dac","Type":"ContainerDied","Data":"97642a3e44417d17a30b89f573939bb7539f03fc959b7b71b073d10c0115553c"}
Oct 30 23:01:12.481769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.481684    4581 scope.go:122] "RemoveContainer" containerID="4e4cd8865cd0d83a0d9d70e64741bcd27ca53265a310f62a09e088a1fca8b60a"
Oct 30 23:01:12.545275 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.545247    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 23:01:12.656613 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.656514    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-kube-api-access-lkl8s\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-kube-api-access-lkl8s\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.656815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.656636    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-certs\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.656815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.656665    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-data\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.656815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.656765    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-istio-token\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-istio-token\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.657002 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.656849    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-socket\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.657002 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.656903    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-credential-socket\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.657002 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.656940    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b658db4-7498-4462-850d-ba2669a04dac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b658db4-7498-4462-850d-ba2669a04dac-istio-podinfo\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.657172 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.657007    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b658db4-7498-4462-850d-ba2669a04dac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b658db4-7498-4462-850d-ba2669a04dac-istiod-ca-cert\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.657172 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.657084    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-envoy\") pod \"2b658db4-7498-4462-850d-ba2669a04dac\" (UID: \"2b658db4-7498-4462-850d-ba2669a04dac\") "
Oct 30 23:01:12.657598 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.657568    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-credential-socket" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:12.657706 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.657621    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-certs" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:12.657795 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.657776    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-socket" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:12.658302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.658178    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.658302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.658191    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b658db4-7498-4462-850d-ba2669a04dac-istiod-ca-cert" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:12.658302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.658204    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.658302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.658261    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.658701 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.658667    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-data" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:12.660636 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.660604    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-istio-token" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:12.661404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.661380    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-kube-api-access-lkl8s" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "kube-api-access-lkl8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:12.661514 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.661443    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-envoy" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:12.661794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.661756    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b658db4-7498-4462-850d-ba2669a04dac-istio-podinfo" pod "2b658db4-7498-4462-850d-ba2669a04dac" (UID: "2b658db4-7498-4462-850d-ba2669a04dac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:01:12.759771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.759739    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.759771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.759770    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.759771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.759780    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b658db4-7498-4462-850d-ba2669a04dac-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.760022 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.759790    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b658db4-7498-4462-850d-ba2669a04dac-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.760022 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.759799    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b658db4-7498-4462-850d-ba2669a04dac-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.760022 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.759808    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkl8s\" (UniqueName: \"kubernetes.io/projected/2b658db4-7498-4462-850d-ba2669a04dac-kube-api-access-lkl8s\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:12.982040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.981953    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"]
Oct 30 23:01:12.982681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.982666    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bff01d2d-904d-4b97-8f67-f928208e138c" containerName="revoke-keys"
Oct 30 23:01:12.982740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.982683    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff01d2d-904d-4b97-8f67-f928208e138c" containerName="revoke-keys"
Oct 30 23:01:12.982740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.982699    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy"
Oct 30 23:01:12.982740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.982703    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy"
Oct 30 23:01:12.982740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.982716    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy"
Oct 30 23:01:12.982740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.982721    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy"
Oct 30 23:01:12.982899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.982850    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy"
Oct 30 23:01:12.982899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.982867    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff01d2d-904d-4b97-8f67-f928208e138c" containerName="revoke-keys"
Oct 30 23:01:12.992167 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:12.992129    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.005966 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.005931    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:01:13.006811 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.006786    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:01:13.006990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.006972    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:01:13.011420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.011394    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:01:13.012555 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.012529    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"]
Oct 30 23:01:13.062602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.062554    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.062789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.062622    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.062789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.062728    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fplrp\" (UniqueName: \"kubernetes.io/projected/65b868a3-c600-4753-a0cf-c3cd71fbffe1-kube-api-access-fplrp\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.164330 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.164293    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.164862 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.164345    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.164862 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.164431    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fplrp\" (UniqueName: \"kubernetes.io/projected/65b868a3-c600-4753-a0cf-c3cd71fbffe1-kube-api-access-fplrp\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.165013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.164974    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.167586 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.167556    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.184453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.184419    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fplrp\" (UniqueName: \"kubernetes.io/projected/65b868a3-c600-4753-a0cf-c3cd71fbffe1-kube-api-access-fplrp\") pod \"maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.315535 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.315444    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:13.492949 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.492911    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm" event={"ID":"2b658db4-7498-4462-850d-ba2669a04dac","Type":"ContainerDied","Data":"d9864da6e53b62c16b89fda45aa248a81716871231f58b64ee3a968b70db55d7"}
Oct 30 23:01:13.492949 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.492942    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"
Oct 30 23:01:13.493177 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.492971    4581 scope.go:122] "RemoveContainer" containerID="97642a3e44417d17a30b89f573939bb7539f03fc959b7b71b073d10c0115553c"
Oct 30 23:01:13.552517 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.550622    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"]
Oct 30 23:01:13.576283 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.576150    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3a7dd-openshift-default-6c66b84dc9-v6hrm"]
Oct 30 23:01:13.651668 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:13.651639    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"]
Oct 30 23:01:13.654072 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:13.654032    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b868a3_c600_4753_a0cf_c3cd71fbffe1.slice/crio-a5301f425f4c6e8ad6c387031cc0ddc281a67e5ec9ef60f2e46ebf73d233c071 WatchSource:0}: Error finding container a5301f425f4c6e8ad6c387031cc0ddc281a67e5ec9ef60f2e46ebf73d233c071: Status 404 returned error can't find the container with id a5301f425f4c6e8ad6c387031cc0ddc281a67e5ec9ef60f2e46ebf73d233c071
Oct 30 23:01:14.101353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:14.101321    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b658db4-7498-4462-850d-ba2669a04dac" path="/var/lib/kubelet/pods/2b658db4-7498-4462-850d-ba2669a04dac/volumes"
Oct 30 23:01:14.507309 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:14.507272    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj" event={"ID":"65b868a3-c600-4753-a0cf-c3cd71fbffe1","Type":"ContainerStarted","Data":"06b4ebf54b9f62c3b1d5878ba2983312326a5a81e2696acd5d4e073ea4f6cee5"}
Oct 30 23:01:14.507824 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:14.507340    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj" event={"ID":"65b868a3-c600-4753-a0cf-c3cd71fbffe1","Type":"ContainerStarted","Data":"a5301f425f4c6e8ad6c387031cc0ddc281a67e5ec9ef60f2e46ebf73d233c071"}
Oct 30 23:01:14.544627 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:14.544572    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj" podStartSLOduration=2.544558151 podStartE2EDuration="2.544558151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:14.540705477 +0000 UTC m=+3258.579644363" watchObservedRunningTime="2026-10-30 23:01:14.544558151 +0000 UTC m=+3258.583497036"
Oct 30 23:01:15.426307 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.422280    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"]
Oct 30 23:01:15.426307 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.423616    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b658db4-7498-4462-850d-ba2669a04dac" containerName="istio-proxy"
Oct 30 23:01:15.436179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.436146    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:15.470244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.465804    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2\" (UID: \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:15.470244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.465852    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945v4\" (UniqueName: \"kubernetes.io/projected/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-kube-api-access-945v4\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2\" (UID: \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:15.470619 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.470284    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"]
Oct 30 23:01:15.485243 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.484624    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-76d54a-dockercfg-29lt2\""
Oct 30 23:01:15.485243 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.485000    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-76d54a\""
Oct 30 23:01:15.509262 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.507697    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"]
Oct 30 23:01:15.512539 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.512344    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:15.523548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.523513    4581 generic.go:356] "Generic (PLEG): container finished" podID="65b868a3-c600-4753-a0cf-c3cd71fbffe1" containerID="06b4ebf54b9f62c3b1d5878ba2983312326a5a81e2696acd5d4e073ea4f6cee5" exitCode=0
Oct 30 23:01:15.523712 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.523627    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj" event={"ID":"65b868a3-c600-4753-a0cf-c3cd71fbffe1","Type":"ContainerDied","Data":"06b4ebf54b9f62c3b1d5878ba2983312326a5a81e2696acd5d4e073ea4f6cee5"}
Oct 30 23:01:15.533860 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.533828    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"]
Oct 30 23:01:15.567974 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.567922    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668ch\" (UniqueName: \"kubernetes.io/projected/84821dee-7355-48ea-b90e-c2b8efb7ef34-kube-api-access-668ch\") pod \"payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt\" (UID: \"84821dee-7355-48ea-b90e-c2b8efb7ef34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:15.568321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.568262    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2\" (UID: \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:15.568321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.568306    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945v4\" (UniqueName: \"kubernetes.io/projected/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-kube-api-access-945v4\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2\" (UID: \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:15.568471 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.568370    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84821dee-7355-48ea-b90e-c2b8efb7ef34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt\" (UID: \"84821dee-7355-48ea-b90e-c2b8efb7ef34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:15.569394 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.569366    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2\" (UID: \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:15.586746 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.586718    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-945v4\" (UniqueName: \"kubernetes.io/projected/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-kube-api-access-945v4\") pod \"payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2\" (UID: \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:15.669576 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.669542    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84821dee-7355-48ea-b90e-c2b8efb7ef34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt\" (UID: \"84821dee-7355-48ea-b90e-c2b8efb7ef34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:15.669789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.669627    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668ch\" (UniqueName: \"kubernetes.io/projected/84821dee-7355-48ea-b90e-c2b8efb7ef34-kube-api-access-668ch\") pod \"payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt\" (UID: \"84821dee-7355-48ea-b90e-c2b8efb7ef34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:15.670303 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.670276    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84821dee-7355-48ea-b90e-c2b8efb7ef34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt\" (UID: \"84821dee-7355-48ea-b90e-c2b8efb7ef34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:15.688409 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.688337    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668ch\" (UniqueName: \"kubernetes.io/projected/84821dee-7355-48ea-b90e-c2b8efb7ef34-kube-api-access-668ch\") pod \"payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt\" (UID: \"84821dee-7355-48ea-b90e-c2b8efb7ef34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:15.786136 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.786102    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:15.831482 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:15.831454    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:16.117668 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.117642    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"]
Oct 30 23:01:16.120087 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:16.120058    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa77793_4b17_4cc7_8731_fcfcfc0cec75.slice/crio-7cc212111bd3cf295399a56c731a41743734fd2b4ef8b1ba5148006928c4f259 WatchSource:0}: Error finding container 7cc212111bd3cf295399a56c731a41743734fd2b4ef8b1ba5148006928c4f259: Status 404 returned error can't find the container with id 7cc212111bd3cf295399a56c731a41743734fd2b4ef8b1ba5148006928c4f259
Oct 30 23:01:16.196489 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.196459    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"]
Oct 30 23:01:16.216980 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:16.216947    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84821dee_7355_48ea_b90e_c2b8efb7ef34.slice/crio-815bad73d6c5e12943c224cccf58226188b2fa849a77991fa89a7339f68228ff WatchSource:0}: Error finding container 815bad73d6c5e12943c224cccf58226188b2fa849a77991fa89a7339f68228ff: Status 404 returned error can't find the container with id 815bad73d6c5e12943c224cccf58226188b2fa849a77991fa89a7339f68228ff
Oct 30 23:01:16.535988 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.535897    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt" event={"ID":"84821dee-7355-48ea-b90e-c2b8efb7ef34","Type":"ContainerStarted","Data":"b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d"}
Oct 30 23:01:16.535988 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.535955    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt" event={"ID":"84821dee-7355-48ea-b90e-c2b8efb7ef34","Type":"ContainerStarted","Data":"815bad73d6c5e12943c224cccf58226188b2fa849a77991fa89a7339f68228ff"}
Oct 30 23:01:16.536541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.535993    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:16.537990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.537966    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2" event={"ID":"0aa77793-4b17-4cc7-8731-fcfcfc0cec75","Type":"ContainerStarted","Data":"2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b"}
Oct 30 23:01:16.538118 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.538007    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2" event={"ID":"0aa77793-4b17-4cc7-8731-fcfcfc0cec75","Type":"ContainerStarted","Data":"7cc212111bd3cf295399a56c731a41743734fd2b4ef8b1ba5148006928c4f259"}
Oct 30 23:01:16.538205 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.538178    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:16.580148 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.580077    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt" podStartSLOduration=1.58005588 podStartE2EDuration="1.58005588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:16.571857266 +0000 UTC m=+3260.610796155" watchObservedRunningTime="2026-10-30 23:01:16.58005588 +0000 UTC m=+3260.618994768"
Oct 30 23:01:16.617947 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.617878    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2" podStartSLOduration=1.6178544700000002 podStartE2EDuration="1.61785447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:16.612248259 +0000 UTC m=+3260.651187141" watchObservedRunningTime="2026-10-30 23:01:16.61785447 +0000 UTC m=+3260.656793358"
Oct 30 23:01:16.785625 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.785589    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:16.912263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.912184    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-openshift-ca\") pod \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") "
Oct 30 23:01:16.912452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.912309    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b868a3-c600-4753-a0cf-c3cd71fbffe1-kube-api-access-fplrp\" (UniqueName: \"kubernetes.io/projected/65b868a3-c600-4753-a0cf-c3cd71fbffe1-kube-api-access-fplrp\") pod \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") "
Oct 30 23:01:16.912452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.912406    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-xks-ca\") pod \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\" (UID: \"65b868a3-c600-4753-a0cf-c3cd71fbffe1\") "
Oct 30 23:01:16.912604 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.912536    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-openshift-ca" pod "65b868a3-c600-4753-a0cf-c3cd71fbffe1" (UID: "65b868a3-c600-4753-a0cf-c3cd71fbffe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:16.912952 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.912930    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:16.914681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.914654    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-xks-ca" pod "65b868a3-c600-4753-a0cf-c3cd71fbffe1" (UID: "65b868a3-c600-4753-a0cf-c3cd71fbffe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:16.914779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:16.914740    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b868a3-c600-4753-a0cf-c3cd71fbffe1-kube-api-access-fplrp" pod "65b868a3-c600-4753-a0cf-c3cd71fbffe1" (UID: "65b868a3-c600-4753-a0cf-c3cd71fbffe1"). InnerVolumeSpecName "kube-api-access-fplrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:17.014622 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:17.014578    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fplrp\" (UniqueName: \"kubernetes.io/projected/65b868a3-c600-4753-a0cf-c3cd71fbffe1-kube-api-access-fplrp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:17.014622 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:17.014618    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b868a3-c600-4753-a0cf-c3cd71fbffe1-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:17.548684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:17.548655    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"
Oct 30 23:01:17.548684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:17.548678    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj" event={"ID":"65b868a3-c600-4753-a0cf-c3cd71fbffe1","Type":"ContainerDied","Data":"a5301f425f4c6e8ad6c387031cc0ddc281a67e5ec9ef60f2e46ebf73d233c071"}
Oct 30 23:01:17.549147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:17.548731    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5301f425f4c6e8ad6c387031cc0ddc281a67e5ec9ef60f2e46ebf73d233c071"
Oct 30 23:01:21.395248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.394303    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"]
Oct 30 23:01:21.395764 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.395523    4581 cpu_manager.go:414] "containerMap: removing container" podUID="65b868a3-c600-4753-a0cf-c3cd71fbffe1" containerName="revoke-keys"
Oct 30 23:01:21.395764 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.395545    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b868a3-c600-4753-a0cf-c3cd71fbffe1" containerName="revoke-keys"
Oct 30 23:01:21.395764 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.395726    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b868a3-c600-4753-a0cf-c3cd71fbffe1" containerName="revoke-keys"
Oct 30 23:01:21.501402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.501342    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"]
Oct 30 23:01:21.501689 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.501477    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:21.527186 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.527146    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f94561-dockercfg-nh4xn\""
Oct 30 23:01:21.527399 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.527200    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f94561\""
Oct 30 23:01:21.554290 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.552100    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vmm\" (UniqueName: \"kubernetes.io/projected/0e4c056b-ec3b-4201-8115-045031025b1b-kube-api-access-d4vmm\") pod \"payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5\" (UID: \"0e4c056b-ec3b-4201-8115-045031025b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:21.554290 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.552262    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4c056b-ec3b-4201-8115-045031025b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5\" (UID: \"0e4c056b-ec3b-4201-8115-045031025b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:21.654174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.654063    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vmm\" (UniqueName: \"kubernetes.io/projected/0e4c056b-ec3b-4201-8115-045031025b1b-kube-api-access-d4vmm\") pod \"payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5\" (UID: \"0e4c056b-ec3b-4201-8115-045031025b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:21.654174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.654163    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4c056b-ec3b-4201-8115-045031025b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5\" (UID: \"0e4c056b-ec3b-4201-8115-045031025b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:21.655087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.655064    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4c056b-ec3b-4201-8115-045031025b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5\" (UID: \"0e4c056b-ec3b-4201-8115-045031025b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:21.672180 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.672144    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vmm\" (UniqueName: \"kubernetes.io/projected/0e4c056b-ec3b-4201-8115-045031025b1b-kube-api-access-d4vmm\") pod \"payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5\" (UID: \"0e4c056b-ec3b-4201-8115-045031025b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:21.835983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:21.835950    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:22.162729 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:22.162696    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4c056b_ec3b_4201_8115_045031025b1b.slice/crio-534bfb9d089c8ca41266ba040c12aeff1dc46c55f24ffa5009dcd577996382ee WatchSource:0}: Error finding container 534bfb9d089c8ca41266ba040c12aeff1dc46c55f24ffa5009dcd577996382ee: Status 404 returned error can't find the container with id 534bfb9d089c8ca41266ba040c12aeff1dc46c55f24ffa5009dcd577996382ee
Oct 30 23:01:22.163257 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:22.163119    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"]
Oct 30 23:01:22.612895 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:22.612859    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5" event={"ID":"0e4c056b-ec3b-4201-8115-045031025b1b","Type":"ContainerStarted","Data":"c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d"}
Oct 30 23:01:22.613381 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:22.612921    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5" event={"ID":"0e4c056b-ec3b-4201-8115-045031025b1b","Type":"ContainerStarted","Data":"534bfb9d089c8ca41266ba040c12aeff1dc46c55f24ffa5009dcd577996382ee"}
Oct 30 23:01:22.613381 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:22.612952    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:22.648427 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:22.648366    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5" podStartSLOduration=1.648347411 podStartE2EDuration="1.648347411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:22.647685216 +0000 UTC m=+3266.686624103" watchObservedRunningTime="2026-10-30 23:01:22.648347411 +0000 UTC m=+3266.687286302"
Oct 30 23:01:27.551487 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:27.551450    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:01:27.552539 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:27.551591    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:01:33.625909 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.625872    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:01:33.879181 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.879092    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"]
Oct 30 23:01:33.883908 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.883885    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:33.898715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.898691    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:01:33.898912 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.898893    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:01:33.899524 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.899509    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:01:33.899636 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.899622    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:01:33.908257 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.906257    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"]
Oct 30 23:01:33.978127 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.978093    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:33.978351 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.978235    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:33.978351 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:33.978269    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2nd\" (UniqueName: \"kubernetes.io/projected/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-kube-api-access-4p2nd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:34.079240 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.079181    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2nd\" (UniqueName: \"kubernetes.io/projected/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-kube-api-access-4p2nd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:34.079539 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.079517    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:34.079658 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.079643    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:34.080527 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.080416    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:34.082403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.082366    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:34.099706 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.099680    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2nd\" (UniqueName: \"kubernetes.io/projected/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-kube-api-access-4p2nd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:34.198454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.198362    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:34.511130 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.511096    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"]
Oct 30 23:01:34.512814 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:34.512783    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cfb32f_3e3d_47c4_b5b3_0509e8a3f0e5.slice/crio-60231e1c3f9a72babfc40e014c4fbb08696da51c13b513b08e691d216cf19efe WatchSource:0}: Error finding container 60231e1c3f9a72babfc40e014c4fbb08696da51c13b513b08e691d216cf19efe: Status 404 returned error can't find the container with id 60231e1c3f9a72babfc40e014c4fbb08696da51c13b513b08e691d216cf19efe
Oct 30 23:01:34.749063 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:34.749026    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm" event={"ID":"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5","Type":"ContainerStarted","Data":"60231e1c3f9a72babfc40e014c4fbb08696da51c13b513b08e691d216cf19efe"}
Oct 30 23:01:35.760674 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:35.760644    4581 generic.go:356] "Generic (PLEG): container finished" podID="d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5" containerID="5d0e3d379f7aaa32bb2b5ff10e014d90aaa657c7325b8ab2ad445b8e75deff05" exitCode=0
Oct 30 23:01:35.761139 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:35.760728    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm" event={"ID":"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5","Type":"ContainerDied","Data":"5d0e3d379f7aaa32bb2b5ff10e014d90aaa657c7325b8ab2ad445b8e75deff05"}
Oct 30 23:01:36.524296 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.523897    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"]
Oct 30 23:01:36.580241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.566811    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:36.580241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.570587    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"]
Oct 30 23:01:36.607270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.603024    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cdfa86\""
Oct 30 23:01:36.607270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.603150    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cdfa86-dockercfg-xpvmj\""
Oct 30 23:01:36.762756 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.762657    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cae1609-18f2-422c-81b5-cc7e850ccc0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d\" (UID: \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:36.763377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.762776    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9s9\" (UniqueName: \"kubernetes.io/projected/0cae1609-18f2-422c-81b5-cc7e850ccc0c-kube-api-access-sz9s9\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d\" (UID: \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:36.865669 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.864505    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cae1609-18f2-422c-81b5-cc7e850ccc0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d\" (UID: \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:36.865669 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.864564    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9s9\" (UniqueName: \"kubernetes.io/projected/0cae1609-18f2-422c-81b5-cc7e850ccc0c-kube-api-access-sz9s9\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d\" (UID: \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:36.874836 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.865973    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cae1609-18f2-422c-81b5-cc7e850ccc0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d\" (UID: \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:36.893680 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.892683    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9s9\" (UniqueName: \"kubernetes.io/projected/0cae1609-18f2-422c-81b5-cc7e850ccc0c-kube-api-access-sz9s9\") pod \"payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d\" (UID: \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:36.936837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.936805    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:36.988114 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:36.988089    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:37.168518 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.168484    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-openshift-ca\") pod \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") "
Oct 30 23:01:37.168775 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.168757    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-kube-api-access-4p2nd\" (UniqueName: \"kubernetes.io/projected/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-kube-api-access-4p2nd\") pod \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") "
Oct 30 23:01:37.168850 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.168834    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-xks-ca\") pod \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\" (UID: \"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5\") "
Oct 30 23:01:37.168904 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.168851    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-openshift-ca" pod "d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5" (UID: "d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:37.169488 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.169462    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:37.171091 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.171061    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-xks-ca" pod "d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5" (UID: "d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:37.171091 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.171068    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-kube-api-access-4p2nd" pod "d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5" (UID: "d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5"). InnerVolumeSpecName "kube-api-access-4p2nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:37.261909 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.261865    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"]
Oct 30 23:01:37.262845 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:37.262818    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cae1609_18f2_422c_81b5_cc7e850ccc0c.slice/crio-42c00db7aeadef06f5733b46f5e02de390cd4981b87e8ecdd0bd61a44d361296 WatchSource:0}: Error finding container 42c00db7aeadef06f5733b46f5e02de390cd4981b87e8ecdd0bd61a44d361296: Status 404 returned error can't find the container with id 42c00db7aeadef06f5733b46f5e02de390cd4981b87e8ecdd0bd61a44d361296
Oct 30 23:01:37.271710 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.271685    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p2nd\" (UniqueName: \"kubernetes.io/projected/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-kube-api-access-4p2nd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:37.271803 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.271717    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:37.795922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.795878    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d" event={"ID":"0cae1609-18f2-422c-81b5-cc7e850ccc0c","Type":"ContainerStarted","Data":"2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493"}
Oct 30 23:01:37.796582 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.795953    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d" event={"ID":"0cae1609-18f2-422c-81b5-cc7e850ccc0c","Type":"ContainerStarted","Data":"42c00db7aeadef06f5733b46f5e02de390cd4981b87e8ecdd0bd61a44d361296"}
Oct 30 23:01:37.796806 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.796782    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:37.797784 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.797760    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"
Oct 30 23:01:37.797932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.797801    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm" event={"ID":"d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5","Type":"ContainerDied","Data":"60231e1c3f9a72babfc40e014c4fbb08696da51c13b513b08e691d216cf19efe"}
Oct 30 23:01:37.797932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.797841    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60231e1c3f9a72babfc40e014c4fbb08696da51c13b513b08e691d216cf19efe"
Oct 30 23:01:37.836086 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:37.836028    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d" podStartSLOduration=1.836009859 podStartE2EDuration="1.836009859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:37.831201854 +0000 UTC m=+3281.870140743" watchObservedRunningTime="2026-10-30 23:01:37.836009859 +0000 UTC m=+3281.874948746"
Oct 30 23:01:38.412749 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.412715    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"]
Oct 30 23:01:38.413664 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.413609    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5" containerName="revoke-keys"
Oct 30 23:01:38.413664 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.413635    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5" containerName="revoke-keys"
Oct 30 23:01:38.413879 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.413862    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5" containerName="revoke-keys"
Oct 30 23:01:38.417779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.417757    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.437760 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.437736    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-bff950-openshift-default-dockercfg-dc4z8\""
Oct 30 23:01:38.450396 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.450367    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"]
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.491916    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3da96f9-b10e-4413-995f-0a3dc89511d4-istiod-ca-cert\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.491982    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-envoy\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.492011    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-credential-socket\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.492090    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-certs\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.492187    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-socket\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.492269    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-podinfo\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.492332    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-token\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.492407    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltdk\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-kube-api-access-8ltdk\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.492655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.492438    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-data\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.648408    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-socket\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.648512    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-podinfo\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.648587    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-token\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.648678    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltdk\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-kube-api-access-8ltdk\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.648715    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-data\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.648813    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3da96f9-b10e-4413-995f-0a3dc89511d4-istiod-ca-cert\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.648867    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-envoy\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.648896    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-credential-socket\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.649006    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-certs\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.649586    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-certs\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.656241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.649893    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-socket\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.657251 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.657201    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-data\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.670850 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.670776    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-envoy\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.671740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.671712    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3da96f9-b10e-4413-995f-0a3dc89511d4-istiod-ca-cert\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.672147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.672128    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-credential-socket\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.698089 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.697749    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-token\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.712896 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.712856    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-podinfo\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.743243 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.724938    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltdk\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-kube-api-access-8ltdk\") pod \"e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") " pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:38.743243 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:38.737462    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:39.103293 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:39.103259    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"]
Oct 30 23:01:39.103924 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:39.103893    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3da96f9_b10e_4413_995f_0a3dc89511d4.slice/crio-16faea1ac361b5d5164b4a9a6815c16eb16c9a1e5c20c69d74e3109b04752ca0 WatchSource:0}: Error finding container 16faea1ac361b5d5164b4a9a6815c16eb16c9a1e5c20c69d74e3109b04752ca0: Status 404 returned error can't find the container with id 16faea1ac361b5d5164b4a9a6815c16eb16c9a1e5c20c69d74e3109b04752ca0
Oct 30 23:01:39.107426 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:39.107388    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:01:39.107522 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:39.107478    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:01:39.107560 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:39.107522    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:01:39.823724 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:39.823685    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" event={"ID":"b3da96f9-b10e-4413-995f-0a3dc89511d4","Type":"ContainerStarted","Data":"196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9"}
Oct 30 23:01:39.823916 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:39.823776    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" event={"ID":"b3da96f9-b10e-4413-995f-0a3dc89511d4","Type":"ContainerStarted","Data":"16faea1ac361b5d5164b4a9a6815c16eb16c9a1e5c20c69d74e3109b04752ca0"}
Oct 30 23:01:39.880498 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:39.880432    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podStartSLOduration=1.8804131800000001 podStartE2EDuration="1.88041318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:39.872886245 +0000 UTC m=+3283.911825130" watchObservedRunningTime="2026-10-30 23:01:39.88041318 +0000 UTC m=+3283.919352068"
Oct 30 23:01:40.744599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:40.744568    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:40.752275 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:40.752203    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:40.820560 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:01:40.820521    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:01:40.833496 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:40.833459    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:40.834542 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:40.834484    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:01:43.239280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.234746    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"]
Oct 30 23:01:43.239912 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.239890    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.265347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.265319    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:01:43.265551 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.265430    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:01:43.273271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.272278    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"]
Oct 30 23:01:43.281391 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.280604    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:01:43.294962 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.294931    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:01:43.396235 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.396179    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfvjk\" (UniqueName: \"kubernetes.io/projected/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-kube-api-access-kfvjk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.396420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.396358    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.396420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.396400    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.497682 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.497585    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.497682 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.497641    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.497916 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.497894    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfvjk\" (UniqueName: \"kubernetes.io/projected/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-kube-api-access-kfvjk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.498626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.498536    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.500401 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.500374    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.515873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.515847    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfvjk\" (UniqueName: \"kubernetes.io/projected/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-kube-api-access-kfvjk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.555821 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.555792    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:43.875868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.875834    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"]
Oct 30 23:01:43.878251 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:43.878196    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd2cd81_751f_4b6f_a51e_fa109644f7f4.slice/crio-bd219ef5fc0b2798a461b2611847430d818163a9ad21d0091b55863c37325403 WatchSource:0}: Error finding container bd219ef5fc0b2798a461b2611847430d818163a9ad21d0091b55863c37325403: Status 404 returned error can't find the container with id bd219ef5fc0b2798a461b2611847430d818163a9ad21d0091b55863c37325403
Oct 30 23:01:43.895291 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:43.895257    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk" event={"ID":"9dd2cd81-751f-4b6f-a51e-fa109644f7f4","Type":"ContainerStarted","Data":"bd219ef5fc0b2798a461b2611847430d818163a9ad21d0091b55863c37325403"}
Oct 30 23:01:44.907759 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:44.907723    4581 generic.go:356] "Generic (PLEG): container finished" podID="9dd2cd81-751f-4b6f-a51e-fa109644f7f4" containerID="2b01c9075fff44d2fcd3c01445cbf1cbab2cbb71c5c929d2d35b4dfdb8f0f2c9" exitCode=0
Oct 30 23:01:44.908179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:44.907800    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk" event={"ID":"9dd2cd81-751f-4b6f-a51e-fa109644f7f4","Type":"ContainerDied","Data":"2b01c9075fff44d2fcd3c01445cbf1cbab2cbb71c5c929d2d35b4dfdb8f0f2c9"}
Oct 30 23:01:46.246948 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.246883    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:46.268358 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.268323    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-xks-ca\") pod \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") "
Oct 30 23:01:46.268535 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.268400    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-kube-api-access-kfvjk\" (UniqueName: \"kubernetes.io/projected/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-kube-api-access-kfvjk\") pod \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") "
Oct 30 23:01:46.268535 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.268441    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-openshift-ca\") pod \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\" (UID: \"9dd2cd81-751f-4b6f-a51e-fa109644f7f4\") "
Oct 30 23:01:46.268855 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.268823    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-openshift-ca" pod "9dd2cd81-751f-4b6f-a51e-fa109644f7f4" (UID: "9dd2cd81-751f-4b6f-a51e-fa109644f7f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:46.269202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.269179    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:46.270809 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.270784    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-xks-ca" pod "9dd2cd81-751f-4b6f-a51e-fa109644f7f4" (UID: "9dd2cd81-751f-4b6f-a51e-fa109644f7f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:46.271207 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.271178    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-kube-api-access-kfvjk" pod "9dd2cd81-751f-4b6f-a51e-fa109644f7f4" (UID: "9dd2cd81-751f-4b6f-a51e-fa109644f7f4"). InnerVolumeSpecName "kube-api-access-kfvjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:46.370476 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.370442    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:46.370476 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.370475    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfvjk\" (UniqueName: \"kubernetes.io/projected/9dd2cd81-751f-4b6f-a51e-fa109644f7f4-kube-api-access-kfvjk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:46.978132 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.978102    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"
Oct 30 23:01:46.978132 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.978114    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk" event={"ID":"9dd2cd81-751f-4b6f-a51e-fa109644f7f4","Type":"ContainerDied","Data":"bd219ef5fc0b2798a461b2611847430d818163a9ad21d0091b55863c37325403"}
Oct 30 23:01:46.978374 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:46.978162    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd219ef5fc0b2798a461b2611847430d818163a9ad21d0091b55863c37325403"
Oct 30 23:01:48.815623 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:48.815592    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:01:51.260935 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.260893    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"]
Oct 30 23:01:51.261690 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.261672    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9dd2cd81-751f-4b6f-a51e-fa109644f7f4" containerName="revoke-keys"
Oct 30 23:01:51.261690 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.261689    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd2cd81-751f-4b6f-a51e-fa109644f7f4" containerName="revoke-keys"
Oct 30 23:01:51.261833 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.261805    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd2cd81-751f-4b6f-a51e-fa109644f7f4" containerName="revoke-keys"
Oct 30 23:01:51.266933 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.266910    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.285261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.285208    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"]
Oct 30 23:01:51.289020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.288992    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:01:51.289020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.288997    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:01:51.289897 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.289875    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:01:51.289897 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.289890    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:01:51.359862 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.359822    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.360079 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.360054    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlvv2\" (UniqueName: \"kubernetes.io/projected/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-kube-api-access-rlvv2\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.360157 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.360121    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.461599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.461561    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.461819 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.461710    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlvv2\" (UniqueName: \"kubernetes.io/projected/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-kube-api-access-rlvv2\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.461819 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.461755    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.462647 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.462615    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.464190 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.464157    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.479465 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.479429    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlvv2\" (UniqueName: \"kubernetes.io/projected/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-kube-api-access-rlvv2\") pod \"maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.582469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.582390    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:51.907994 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:51.907947    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"]
Oct 30 23:01:51.909686 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:01:51.909655    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b95030_87ba_495f_ba9b_9c4c9ac4af5a.slice/crio-a4e4b76a6b1f569de90c80f95830d590e3137d2cf9490c5c945f841041ee943c WatchSource:0}: Error finding container a4e4b76a6b1f569de90c80f95830d590e3137d2cf9490c5c945f841041ee943c: Status 404 returned error can't find the container with id a4e4b76a6b1f569de90c80f95830d590e3137d2cf9490c5c945f841041ee943c
Oct 30 23:01:52.047481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:52.047446    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm" event={"ID":"89b95030-87ba-495f-ba9b-9c4c9ac4af5a","Type":"ContainerStarted","Data":"a4e4b76a6b1f569de90c80f95830d590e3137d2cf9490c5c945f841041ee943c"}
Oct 30 23:01:53.059521 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:53.059487    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm" event={"ID":"89b95030-87ba-495f-ba9b-9c4c9ac4af5a","Type":"ContainerStarted","Data":"5e1b48a2ae5caf061cf6d266bfe1e03172a9021dc282314afb419ae1a7a184a5"}
Oct 30 23:01:53.098165 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:53.098112    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm" podStartSLOduration=2.098096603 podStartE2EDuration="2.098096603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:53.093963847 +0000 UTC m=+3297.132902732" watchObservedRunningTime="2026-10-30 23:01:53.098096603 +0000 UTC m=+3297.137035489"
Oct 30 23:01:53.647135 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:53.647100    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"]
Oct 30 23:01:54.072522 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:54.072420    4581 generic.go:356] "Generic (PLEG): container finished" podID="89b95030-87ba-495f-ba9b-9c4c9ac4af5a" containerID="5e1b48a2ae5caf061cf6d266bfe1e03172a9021dc282314afb419ae1a7a184a5" exitCode=0
Oct 30 23:01:54.072522 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:54.072456    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm" event={"ID":"89b95030-87ba-495f-ba9b-9c4c9ac4af5a","Type":"ContainerDied","Data":"5e1b48a2ae5caf061cf6d266bfe1e03172a9021dc282314afb419ae1a7a184a5"}
Oct 30 23:01:55.290832 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.290799    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:55.424857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.424822    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-xks-ca\") pod \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") "
Oct 30 23:01:55.425063 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.424882    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-kube-api-access-rlvv2\" (UniqueName: \"kubernetes.io/projected/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-kube-api-access-rlvv2\") pod \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") "
Oct 30 23:01:55.425063 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.425008    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-openshift-ca\") pod \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\" (UID: \"89b95030-87ba-495f-ba9b-9c4c9ac4af5a\") "
Oct 30 23:01:55.425395 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.425369    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-openshift-ca" pod "89b95030-87ba-495f-ba9b-9c4c9ac4af5a" (UID: "89b95030-87ba-495f-ba9b-9c4c9ac4af5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:55.425814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.425754    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:55.427331 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.427299    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-xks-ca" pod "89b95030-87ba-495f-ba9b-9c4c9ac4af5a" (UID: "89b95030-87ba-495f-ba9b-9c4c9ac4af5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:55.427331 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.427304    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-kube-api-access-rlvv2" pod "89b95030-87ba-495f-ba9b-9c4c9ac4af5a" (UID: "89b95030-87ba-495f-ba9b-9c4c9ac4af5a"). InnerVolumeSpecName "kube-api-access-rlvv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:55.527637 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.527594    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlvv2\" (UniqueName: \"kubernetes.io/projected/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-kube-api-access-rlvv2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:55.527637 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.527627    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b95030-87ba-495f-ba9b-9c4c9ac4af5a-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:55.978558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.978516    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"]
Oct 30 23:01:55.978820 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.978792    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls" podUID="9142b111-3a39-4408-9784-52eda5706504" containerName="payload-processing" containerID="cri-o://ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2" gracePeriod=30
Oct 30 23:01:55.999154 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.999122    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"]
Oct 30 23:01:55.999422 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:55.999397    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw" podUID="ae303102-96fd-484c-9bbf-ef20ed64ff25" containerName="payload-pre-processing" containerID="cri-o://2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b" gracePeriod=30
Oct 30 23:01:56.098382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.097649    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"
Oct 30 23:01:56.110033 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.110003    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm" event={"ID":"89b95030-87ba-495f-ba9b-9c4c9ac4af5a","Type":"ContainerDied","Data":"a4e4b76a6b1f569de90c80f95830d590e3137d2cf9490c5c945f841041ee943c"}
Oct 30 23:01:56.110033 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.110036    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e4b76a6b1f569de90c80f95830d590e3137d2cf9490c5c945f841041ee943c"
Oct 30 23:01:56.528180 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.527882    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 23:01:56.563565 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.563539    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 23:01:56.591163 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.591096    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"]
Oct 30 23:01:56.597061 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.597002    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a60e80-h98rj"]
Oct 30 23:01:56.629326 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.629296    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae303102-96fd-484c-9bbf-ef20ed64ff25-kube-api-access-286fb\" (UniqueName: \"kubernetes.io/projected/ae303102-96fd-484c-9bbf-ef20ed64ff25-kube-api-access-286fb\") pod \"ae303102-96fd-484c-9bbf-ef20ed64ff25\" (UID: \"ae303102-96fd-484c-9bbf-ef20ed64ff25\") "
Oct 30 23:01:56.629526 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.629370    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae303102-96fd-484c-9bbf-ef20ed64ff25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae303102-96fd-484c-9bbf-ef20ed64ff25-plugins-config-volume\") pod \"ae303102-96fd-484c-9bbf-ef20ed64ff25\" (UID: \"ae303102-96fd-484c-9bbf-ef20ed64ff25\") "
Oct 30 23:01:56.629526 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.629506    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9142b111-3a39-4408-9784-52eda5706504-kube-api-access-m8bh9\" (UniqueName: \"kubernetes.io/projected/9142b111-3a39-4408-9784-52eda5706504-kube-api-access-m8bh9\") pod \"9142b111-3a39-4408-9784-52eda5706504\" (UID: \"9142b111-3a39-4408-9784-52eda5706504\") "
Oct 30 23:01:56.629653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.629623    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9142b111-3a39-4408-9784-52eda5706504-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142b111-3a39-4408-9784-52eda5706504-plugins-config-volume\") pod \"9142b111-3a39-4408-9784-52eda5706504\" (UID: \"9142b111-3a39-4408-9784-52eda5706504\") "
Oct 30 23:01:56.629853 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.629818    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae303102-96fd-484c-9bbf-ef20ed64ff25-plugins-config-volume" pod "ae303102-96fd-484c-9bbf-ef20ed64ff25" (UID: "ae303102-96fd-484c-9bbf-ef20ed64ff25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:56.630307 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.630278    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9142b111-3a39-4408-9784-52eda5706504-plugins-config-volume" pod "9142b111-3a39-4408-9784-52eda5706504" (UID: "9142b111-3a39-4408-9784-52eda5706504"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:56.630412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.630344    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae303102-96fd-484c-9bbf-ef20ed64ff25-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:56.631841 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.631811    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae303102-96fd-484c-9bbf-ef20ed64ff25-kube-api-access-286fb" pod "ae303102-96fd-484c-9bbf-ef20ed64ff25" (UID: "ae303102-96fd-484c-9bbf-ef20ed64ff25"). InnerVolumeSpecName "kube-api-access-286fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:56.633187 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.633162    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9142b111-3a39-4408-9784-52eda5706504-kube-api-access-m8bh9" pod "9142b111-3a39-4408-9784-52eda5706504" (UID: "9142b111-3a39-4408-9784-52eda5706504"). InnerVolumeSpecName "kube-api-access-m8bh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:56.731624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.731596    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-286fb\" (UniqueName: \"kubernetes.io/projected/ae303102-96fd-484c-9bbf-ef20ed64ff25-kube-api-access-286fb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:56.731624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.731621    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8bh9\" (UniqueName: \"kubernetes.io/projected/9142b111-3a39-4408-9784-52eda5706504-kube-api-access-m8bh9\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:56.731624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:56.731631    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142b111-3a39-4408-9784-52eda5706504-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:01:57.109608 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.109567    4581 generic.go:356] "Generic (PLEG): container finished" podID="ae303102-96fd-484c-9bbf-ef20ed64ff25" containerID="2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b" exitCode=0
Oct 30 23:01:57.109608 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.109611    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"
Oct 30 23:01:57.109891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.109659    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw" event={"ID":"ae303102-96fd-484c-9bbf-ef20ed64ff25","Type":"ContainerDied","Data":"2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b"}
Oct 30 23:01:57.109891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.109716    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw" event={"ID":"ae303102-96fd-484c-9bbf-ef20ed64ff25","Type":"ContainerDied","Data":"1d57ecd6afd68df2cb8fff8877b52efd9e3fed292d6cf7d74ff20bbfb0c88cf3"}
Oct 30 23:01:57.109891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.109738    4581 scope.go:122] "RemoveContainer" containerID="2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b"
Oct 30 23:01:57.111336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.111309    4581 generic.go:356] "Generic (PLEG): container finished" podID="9142b111-3a39-4408-9784-52eda5706504" containerID="ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2" exitCode=0
Oct 30 23:01:57.111487 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.111346    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"
Oct 30 23:01:57.111487 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.111436    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls" event={"ID":"9142b111-3a39-4408-9784-52eda5706504","Type":"ContainerDied","Data":"ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2"}
Oct 30 23:01:57.111603 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.111503    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls" event={"ID":"9142b111-3a39-4408-9784-52eda5706504","Type":"ContainerDied","Data":"c62503d7a24f2c67212a0943d1fad4bf26d5de93eacb56ea59c0b28b45609a8d"}
Oct 30 23:01:57.126112 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.126089    4581 scope.go:122] "RemoveContainer" containerID="2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b"
Oct 30 23:01:57.126472 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:01:57.126451    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b\": container with ID starting with 2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b not found: ID does not exist" containerID="2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b"
Oct 30 23:01:57.126570 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.126481    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b"} err="failed to get container status \"2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b\": rpc error: code = NotFound desc = could not find container \"2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b\": container with ID starting with 2a532b0a7b64edb4596ccbed64eda0b9fdc39103ba29a00f01183dbe6959243b not found: ID does not exist"
Oct 30 23:01:57.126570 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.126498    4581 scope.go:122] "RemoveContainer" containerID="ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2"
Oct 30 23:01:57.139936 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.139914    4581 scope.go:122] "RemoveContainer" containerID="ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2"
Oct 30 23:01:57.140253 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:01:57.140205    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2\": container with ID starting with ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2 not found: ID does not exist" containerID="ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2"
Oct 30 23:01:57.140322 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.140262    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2"} err="failed to get container status \"ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2\": rpc error: code = NotFound desc = could not find container \"ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2\": container with ID starting with ade937e5451a1f66ad62d887d421b415438f2b3cfc391d8775654ea3623e51d2 not found: ID does not exist"
Oct 30 23:01:57.174663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.174128    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"]
Oct 30 23:01:57.187933 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.187892    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a60e80-86d69656d4-9r2ls"]
Oct 30 23:01:57.203806 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.203775    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"]
Oct 30 23:01:57.210651 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.210623    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a60e80-5db87c97f9-7rjpw"]
Oct 30 23:01:57.485245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.485132    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:01:57.491742 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:57.491719    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:01:58.099085 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:58.099051    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b868a3-c600-4753-a0cf-c3cd71fbffe1" path="/var/lib/kubelet/pods/65b868a3-c600-4753-a0cf-c3cd71fbffe1/volumes"
Oct 30 23:01:58.099577 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:58.099560    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9142b111-3a39-4408-9784-52eda5706504" path="/var/lib/kubelet/pods/9142b111-3a39-4408-9784-52eda5706504/volumes"
Oct 30 23:01:58.099947 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:58.099933    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae303102-96fd-484c-9bbf-ef20ed64ff25" path="/var/lib/kubelet/pods/ae303102-96fd-484c-9bbf-ef20ed64ff25/volumes"
Oct 30 23:01:58.647730 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:01:58.647687    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj" podUID="3e043128-f90a-469a-ae0c-a09a7e9a0654" containerName="payload-processing" containerID="cri-o://b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988" gracePeriod=25
Oct 30 23:02:02.381136 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.381104    4581 scope.go:122] "RemoveContainer" containerID="e5643a05b7e1daae68a6fb18d460733d74e6db3020d218d604dd437e78351f82"
Oct 30 23:02:02.392743 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.392716    4581 scope.go:122] "RemoveContainer" containerID="d93698fcf578c70d2edfff7ccb1f03d58c60eb65b0959e17a238328bddbdc8e6"
Oct 30 23:02:02.404204 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.404180    4581 scope.go:122] "RemoveContainer" containerID="18995c2b7f2dfb2f4c4b5e47110dc353bd74c2509b40cf626db8e6ff05b4a706"
Oct 30 23:02:02.415995 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.415969    4581 scope.go:122] "RemoveContainer" containerID="d01fc5dc710310fe7178d504eaacc5bcfff79a0800ab6decf3840dbdb9c7ba4c"
Oct 30 23:02:02.427480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.427452    4581 scope.go:122] "RemoveContainer" containerID="296eef6b7fd6244efcd1c439634f689fc2d04ba89e242a1972eaeed4af5b1048"
Oct 30 23:02:02.438144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.438125    4581 scope.go:122] "RemoveContainer" containerID="70e286f583eef379bad5e48f26ad7ddf0efd61e707db33f83d303e386cd35f69"
Oct 30 23:02:02.449019 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.449000    4581 scope.go:122] "RemoveContainer" containerID="77f550f86d3fc207e7f9671e557180e5f3fd5ad386c8496f8feaf6169c57f8f6"
Oct 30 23:02:02.460798 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.460777    4581 scope.go:122] "RemoveContainer" containerID="6a9fd7acf6b5c68823fb404b60df1f6b626b2337b7239e9092b34653250c4c8c"
Oct 30 23:02:02.473477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.473451    4581 scope.go:122] "RemoveContainer" containerID="fea969036d9d607c39dd602ef6b3053137670b2cb455a346717dc5e517f52f0a"
Oct 30 23:02:02.485203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:02.485179    4581 scope.go:122] "RemoveContainer" containerID="03167dbbeb2cda87142ca16930a3e7ff2e59ea0c8d12d5f238a905783e226e4e"
Oct 30 23:02:03.464681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:03.464646    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"]
Oct 30 23:02:03.465512 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:03.465475    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy" containerID="cri-o://a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c" gracePeriod=30
Oct 30 23:02:04.085995 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.085959    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"]
Oct 30 23:02:04.087083 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087052    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9142b111-3a39-4408-9784-52eda5706504" containerName="payload-processing"
Oct 30 23:02:04.087083 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087076    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9142b111-3a39-4408-9784-52eda5706504" containerName="payload-processing"
Oct 30 23:02:04.087651 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087153    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ae303102-96fd-484c-9bbf-ef20ed64ff25" containerName="payload-pre-processing"
Oct 30 23:02:04.087651 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087161    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae303102-96fd-484c-9bbf-ef20ed64ff25" containerName="payload-pre-processing"
Oct 30 23:02:04.087651 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087179    4581 cpu_manager.go:414] "containerMap: removing container" podUID="89b95030-87ba-495f-ba9b-9c4c9ac4af5a" containerName="revoke-keys"
Oct 30 23:02:04.087651 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087186    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b95030-87ba-495f-ba9b-9c4c9ac4af5a" containerName="revoke-keys"
Oct 30 23:02:04.087651 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087385    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae303102-96fd-484c-9bbf-ef20ed64ff25" containerName="payload-pre-processing"
Oct 30 23:02:04.087651 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087398    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b95030-87ba-495f-ba9b-9c4c9ac4af5a" containerName="revoke-keys"
Oct 30 23:02:04.087651 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.087411    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9142b111-3a39-4408-9784-52eda5706504" containerName="payload-processing"
Oct 30 23:02:04.094284 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.094256    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.110951 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.110921    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"]
Oct 30 23:02:04.114006 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.113960    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:02:04.114170 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.114010    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:02:04.114170 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.114065    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:02:04.117282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.117263    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:02:04.165753 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.165721    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.165963 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.165811    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf8xn\" (UniqueName: \"kubernetes.io/projected/074eaa00-c73a-448d-8c3c-8fe99db910e9-kube-api-access-hf8xn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.166140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.166068    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.267376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.267337    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.267376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.267390    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf8xn\" (UniqueName: \"kubernetes.io/projected/074eaa00-c73a-448d-8c3c-8fe99db910e9-kube-api-access-hf8xn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.267677 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.267506    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.268330 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.268296    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.270071 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.270044    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.285115 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.285090    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf8xn\" (UniqueName: \"kubernetes.io/projected/074eaa00-c73a-448d-8c3c-8fe99db910e9-kube-api-access-hf8xn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.412149 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.412115    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:04.741909 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.741879    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"]
Oct 30 23:02:04.743606 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:02:04.743576    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074eaa00_c73a_448d_8c3c_8fe99db910e9.slice/crio-c9f944bbc97b58378e277a332f3681a2662dec7049200a883a20a88765d7bec3 WatchSource:0}: Error finding container c9f944bbc97b58378e277a332f3681a2662dec7049200a883a20a88765d7bec3: Status 404 returned error can't find the container with id c9f944bbc97b58378e277a332f3681a2662dec7049200a883a20a88765d7bec3
Oct 30 23:02:04.821424 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.821389    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:02:04.821593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:04.821442    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:02:05.199885 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:05.199842    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz" event={"ID":"074eaa00-c73a-448d-8c3c-8fe99db910e9","Type":"ContainerStarted","Data":"389a15353a1e13219db70b18a307d5bbc8e3c149467e82ca75fad2d94d67f352"}
Oct 30 23:02:05.200081 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:05.199894    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz" event={"ID":"074eaa00-c73a-448d-8c3c-8fe99db910e9","Type":"ContainerStarted","Data":"c9f944bbc97b58378e277a332f3681a2662dec7049200a883a20a88765d7bec3"}
Oct 30 23:02:05.239060 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:05.238991    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz" podStartSLOduration=1.238970686 podStartE2EDuration="1.238970686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:05.235258989 +0000 UTC m=+3309.274197874" watchObservedRunningTime="2026-10-30 23:02:05.238970686 +0000 UTC m=+3309.277909571"
Oct 30 23:02:06.210706 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:06.210672    4581 generic.go:356] "Generic (PLEG): container finished" podID="074eaa00-c73a-448d-8c3c-8fe99db910e9" containerID="389a15353a1e13219db70b18a307d5bbc8e3c149467e82ca75fad2d94d67f352" exitCode=0
Oct 30 23:02:06.211133 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:06.210801    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz" event={"ID":"074eaa00-c73a-448d-8c3c-8fe99db910e9","Type":"ContainerDied","Data":"389a15353a1e13219db70b18a307d5bbc8e3c149467e82ca75fad2d94d67f352"}
Oct 30 23:02:07.430462 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.430424    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:07.528077 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.527987    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074eaa00-c73a-448d-8c3c-8fe99db910e9-kube-api-access-hf8xn\" (UniqueName: \"kubernetes.io/projected/074eaa00-c73a-448d-8c3c-8fe99db910e9-kube-api-access-hf8xn\") pod \"074eaa00-c73a-448d-8c3c-8fe99db910e9\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") "
Oct 30 23:02:07.528253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.528128    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-xks-ca\") pod \"074eaa00-c73a-448d-8c3c-8fe99db910e9\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") "
Oct 30 23:02:07.528253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.528165    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-openshift-ca\") pod \"074eaa00-c73a-448d-8c3c-8fe99db910e9\" (UID: \"074eaa00-c73a-448d-8c3c-8fe99db910e9\") "
Oct 30 23:02:07.528616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.528575    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-openshift-ca" pod "074eaa00-c73a-448d-8c3c-8fe99db910e9" (UID: "074eaa00-c73a-448d-8c3c-8fe99db910e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:07.528995 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.528971    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:07.530307 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.530285    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074eaa00-c73a-448d-8c3c-8fe99db910e9-kube-api-access-hf8xn" pod "074eaa00-c73a-448d-8c3c-8fe99db910e9" (UID: "074eaa00-c73a-448d-8c3c-8fe99db910e9"). InnerVolumeSpecName "kube-api-access-hf8xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:07.530526 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.530505    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-xks-ca" pod "074eaa00-c73a-448d-8c3c-8fe99db910e9" (UID: "074eaa00-c73a-448d-8c3c-8fe99db910e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:07.630527 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.630489    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf8xn\" (UniqueName: \"kubernetes.io/projected/074eaa00-c73a-448d-8c3c-8fe99db910e9-kube-api-access-hf8xn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:07.630527 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:07.630518    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074eaa00-c73a-448d-8c3c-8fe99db910e9-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.229925 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.229890    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"
Oct 30 23:02:08.230125 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.229918    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz" event={"ID":"074eaa00-c73a-448d-8c3c-8fe99db910e9","Type":"ContainerDied","Data":"c9f944bbc97b58378e277a332f3681a2662dec7049200a883a20a88765d7bec3"}
Oct 30 23:02:08.230125 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.229971    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9f944bbc97b58378e277a332f3681a2662dec7049200a883a20a88765d7bec3"
Oct 30 23:02:08.369068 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.368972    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj" podUID="3e043128-f90a-469a-ae0c-a09a7e9a0654" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:08.779405 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.779375    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 23:02:08.847160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847124    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-envoy\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847375 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847236    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-certs\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847375 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847303    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-credential-socket\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847375 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847340    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-podinfo\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847442    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istiod-ca-cert\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847512    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-socket\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847568    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-token\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-token\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847616    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-data\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847889 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847776    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-kube-api-access-jn547\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-kube-api-access-jn547\") pod \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\" (UID: \"2f0b4562-7c42-4605-ae23-06de1c2a5cfd\") "
Oct 30 23:02:08.847889 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847799    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-socket" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:08.847889 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847835    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-certs" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:08.848060 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.847880    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istiod-ca-cert" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:08.848060 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.848015    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-data" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:08.848424 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.848400    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-credential-socket" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:08.848579 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.848546    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.848579 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.848563    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.848696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.848575    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.848696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.848601    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.848696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.848616    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.851174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.851142    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-kube-api-access-jn547" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "kube-api-access-jn547". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:08.858641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.858596    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-podinfo" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:02:08.858918 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.858898    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-envoy" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:08.860535 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.860497    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-token" pod "2f0b4562-7c42-4605-ae23-06de1c2a5cfd" (UID: "2f0b4562-7c42-4605-ae23-06de1c2a5cfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:08.949928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.949827    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.949928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.949862    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn547\" (UniqueName: \"kubernetes.io/projected/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-kube-api-access-jn547\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.949928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.949875    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:08.949928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:08.949889    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0b4562-7c42-4605-ae23-06de1c2a5cfd-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:09.242541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.242441    4581 generic.go:356] "Generic (PLEG): container finished" podID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerID="a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c" exitCode=0
Oct 30 23:02:09.242541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.242503    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"
Oct 30 23:02:09.242541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.242523    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" event={"ID":"2f0b4562-7c42-4605-ae23-06de1c2a5cfd","Type":"ContainerDied","Data":"a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c"}
Oct 30 23:02:09.242831 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.242593    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt" event={"ID":"2f0b4562-7c42-4605-ae23-06de1c2a5cfd","Type":"ContainerDied","Data":"4764df4a9079d99b62e8f9c3cc2f2a274df2823d394bd9daaa173455b819d173"}
Oct 30 23:02:09.242831 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.242622    4581 scope.go:122] "RemoveContainer" containerID="a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c"
Oct 30 23:02:09.256918 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.256898    4581 scope.go:122] "RemoveContainer" containerID="f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e"
Oct 30 23:02:09.268886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.268862    4581 scope.go:122] "RemoveContainer" containerID="a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c"
Oct 30 23:02:09.269280 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:09.269260    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c\": container with ID starting with a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c not found: ID does not exist" containerID="a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c"
Oct 30 23:02:09.269361 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.269291    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c"} err="failed to get container status \"a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c\": rpc error: code = NotFound desc = could not find container \"a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c\": container with ID starting with a71b27fbb8421e5a9589132d037f60f3dec8ffd410ca9e763e0b833b8dc0ee5c not found: ID does not exist"
Oct 30 23:02:09.269361 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.269309    4581 scope.go:122] "RemoveContainer" containerID="f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e"
Oct 30 23:02:09.269664 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:09.269635    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e\": container with ID starting with f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e not found: ID does not exist" containerID="f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e"
Oct 30 23:02:09.269842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.269670    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e"} err="failed to get container status \"f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e\": rpc error: code = NotFound desc = could not find container \"f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e\": container with ID starting with f6b85ceee4b0db059c32ee2d5a92809265254139f70b6eb0ebb8ee54b097528e not found: ID does not exist"
Oct 30 23:02:09.291283 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.291248    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"]
Oct 30 23:02:09.296148 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:09.296122    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a60e80-openshift-default-c8f96989c-rvvlt"]
Oct 30 23:02:10.100144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:10.100105    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" path="/var/lib/kubelet/pods/2f0b4562-7c42-4605-ae23-06de1c2a5cfd/volumes"
Oct 30 23:02:10.354489 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:10.354446    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"]
Oct 30 23:02:10.354717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:10.354642    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy" containerID="cri-o://40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673" gracePeriod=30
Oct 30 23:02:10.819376 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:10.819287    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:02:16.096770 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.096738    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 23:02:16.204352 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204253    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-kube-api-access-5d57m\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-kube-api-access-5d57m\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204352 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204308    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-token\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204352 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204354    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-envoy\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204396    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-credential-socket\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204508    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-certs\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204535    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-socket\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204563    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-data\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204634    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-podinfo\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204759    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istiod-ca-cert\") pod \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\" (UID: \"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397\") "
Oct 30 23:02:16.204961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204800    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-certs" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:16.204961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.204826    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-socket" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:16.205182 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.205160    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istiod-ca-cert" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:16.205288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.205191    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-data" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:16.205736 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.205708    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.205862 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.205739    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.205862 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.205753    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.205862 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.205767    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.206030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.205988    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-credential-socket" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:16.207378 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.207348    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-kube-api-access-5d57m" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "kube-api-access-5d57m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:16.207766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.207742    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-podinfo" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:02:16.208289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.208266    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-envoy" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:16.208875 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.208851    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-token" pod "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" (UID: "bdf36b5e-49ce-4a82-bad4-fdfcdca3c397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:16.306820 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.306787    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.306820 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.306819    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d57m\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-kube-api-access-5d57m\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.306820 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.306829    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.307163 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.306838    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.307163 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.306846    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:16.311607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.311579    4581 generic.go:356] "Generic (PLEG): container finished" podID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerID="40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673" exitCode=0
Oct 30 23:02:16.311771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.311640    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"
Oct 30 23:02:16.311771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.311659    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" event={"ID":"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397","Type":"ContainerDied","Data":"40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673"}
Oct 30 23:02:16.311771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.311712    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x" event={"ID":"bdf36b5e-49ce-4a82-bad4-fdfcdca3c397","Type":"ContainerDied","Data":"922c5acbd15f1b75342fa66623d95dc189a84bc183b716b5dde43583ab7cab88"}
Oct 30 23:02:16.311771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.311736    4581 scope.go:122] "RemoveContainer" containerID="40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673"
Oct 30 23:02:16.324480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.324463    4581 scope.go:122] "RemoveContainer" containerID="40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673"
Oct 30 23:02:16.324786 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:16.324769    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673\": container with ID starting with 40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673 not found: ID does not exist" containerID="40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673"
Oct 30 23:02:16.324857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.324793    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673"} err="failed to get container status \"40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673\": rpc error: code = NotFound desc = could not find container \"40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673\": container with ID starting with 40bd0f5b1d68e3e3cbaded3f1e23ec6626ca47673dbd2195573291622ef2e673 not found: ID does not exist"
Oct 30 23:02:16.360057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.360022    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"]
Oct 30 23:02:16.365241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:16.365194    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-80ecb3-openshift-default-75b8ddb949-dr69x"]
Oct 30 23:02:17.737341 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.737309    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-190f6b"]
Oct 30 23:02:17.738060 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738044    4581 cpu_manager.go:414] "containerMap: removing container" podUID="074eaa00-c73a-448d-8c3c-8fe99db910e9" containerName="revoke-keys"
Oct 30 23:02:17.738106 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738065    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074eaa00-c73a-448d-8c3c-8fe99db910e9" containerName="revoke-keys"
Oct 30 23:02:17.738140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738107    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy"
Oct 30 23:02:17.738140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738114    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy"
Oct 30 23:02:17.738140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738121    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy"
Oct 30 23:02:17.738140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738126    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy"
Oct 30 23:02:17.738140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738137    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy"
Oct 30 23:02:17.738347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738141    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy"
Oct 30 23:02:17.738347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738279    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" containerName="istio-proxy"
Oct 30 23:02:17.738347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738290    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy"
Oct 30 23:02:17.738347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738298    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0b4562-7c42-4605-ae23-06de1c2a5cfd" containerName="istio-proxy"
Oct 30 23:02:17.738347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.738313    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="074eaa00-c73a-448d-8c3c-8fe99db910e9" containerName="revoke-keys"
Oct 30 23:02:17.743404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.743382    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-190f6b"
Oct 30 23:02:17.763202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.762898    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xbw52\""
Oct 30 23:02:17.764900 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.764873    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-190f6b"]
Oct 30 23:02:17.928781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:17.928739    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wkd\" (UniqueName: \"kubernetes.io/projected/c154ba4b-0e64-4fc3-bc31-8f392e988153-kube-api-access-w7wkd\") pod \"test-curl-7269-190f6b\" (UID: \"c154ba4b-0e64-4fc3-bc31-8f392e988153\") " pod="opendatahub/test-curl-7269-190f6b"
Oct 30 23:02:18.030184 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.030087    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wkd\" (UniqueName: \"kubernetes.io/projected/c154ba4b-0e64-4fc3-bc31-8f392e988153-kube-api-access-w7wkd\") pod \"test-curl-7269-190f6b\" (UID: \"c154ba4b-0e64-4fc3-bc31-8f392e988153\") " pod="opendatahub/test-curl-7269-190f6b"
Oct 30 23:02:18.047261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.047233    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wkd\" (UniqueName: \"kubernetes.io/projected/c154ba4b-0e64-4fc3-bc31-8f392e988153-kube-api-access-w7wkd\") pod \"test-curl-7269-190f6b\" (UID: \"c154ba4b-0e64-4fc3-bc31-8f392e988153\") " pod="opendatahub/test-curl-7269-190f6b"
Oct 30 23:02:18.059595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.059568    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-190f6b"
Oct 30 23:02:18.113639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.113597    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf36b5e-49ce-4a82-bad4-fdfcdca3c397" path="/var/lib/kubelet/pods/bdf36b5e-49ce-4a82-bad4-fdfcdca3c397/volumes"
Oct 30 23:02:18.432357 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.432315    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj" podUID="3e043128-f90a-469a-ae0c-a09a7e9a0654" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:02:18.516054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.516032    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-190f6b"]
Oct 30 23:02:18.758003 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.757923    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"]
Oct 30 23:02:18.763957 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.763928    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-0ff5b0-m4htz"]
Oct 30 23:02:18.971975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:18.971941    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 23:02:19.018160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.018075    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e043128-f90a-469a-ae0c-a09a7e9a0654-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e043128-f90a-469a-ae0c-a09a7e9a0654-plugins-config-volume\") pod \"3e043128-f90a-469a-ae0c-a09a7e9a0654\" (UID: \"3e043128-f90a-469a-ae0c-a09a7e9a0654\") "
Oct 30 23:02:19.018367 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.018171    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e043128-f90a-469a-ae0c-a09a7e9a0654-kube-api-access-4gjrk\" (UniqueName: \"kubernetes.io/projected/3e043128-f90a-469a-ae0c-a09a7e9a0654-kube-api-access-4gjrk\") pod \"3e043128-f90a-469a-ae0c-a09a7e9a0654\" (UID: \"3e043128-f90a-469a-ae0c-a09a7e9a0654\") "
Oct 30 23:02:19.018556 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.018527    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e043128-f90a-469a-ae0c-a09a7e9a0654-plugins-config-volume" pod "3e043128-f90a-469a-ae0c-a09a7e9a0654" (UID: "3e043128-f90a-469a-ae0c-a09a7e9a0654"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:19.018816 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.018793    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e043128-f90a-469a-ae0c-a09a7e9a0654-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:19.020540 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.020510    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e043128-f90a-469a-ae0c-a09a7e9a0654-kube-api-access-4gjrk" pod "3e043128-f90a-469a-ae0c-a09a7e9a0654" (UID: "3e043128-f90a-469a-ae0c-a09a7e9a0654"). InnerVolumeSpecName "kube-api-access-4gjrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:19.119955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.119914    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gjrk\" (UniqueName: \"kubernetes.io/projected/3e043128-f90a-469a-ae0c-a09a7e9a0654-kube-api-access-4gjrk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:19.495110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.495070    4581 generic.go:356] "Generic (PLEG): container finished" podID="3e043128-f90a-469a-ae0c-a09a7e9a0654" containerID="b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988" exitCode=0
Oct 30 23:02:19.495313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.495126    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"
Oct 30 23:02:19.495313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.495133    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj" event={"ID":"3e043128-f90a-469a-ae0c-a09a7e9a0654","Type":"ContainerDied","Data":"b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988"}
Oct 30 23:02:19.495313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.495200    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj" event={"ID":"3e043128-f90a-469a-ae0c-a09a7e9a0654","Type":"ContainerDied","Data":"60f0d870aca14ed9e7fc16e8273c96cd0534329962d110153545b7015f3f1da3"}
Oct 30 23:02:19.495313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.495270    4581 scope.go:122] "RemoveContainer" containerID="b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988"
Oct 30 23:02:19.497398 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.497371    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-190f6b" event={"ID":"c154ba4b-0e64-4fc3-bc31-8f392e988153","Type":"ContainerStarted","Data":"73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571"}
Oct 30 23:02:19.497518 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.497425    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-190f6b" event={"ID":"c154ba4b-0e64-4fc3-bc31-8f392e988153","Type":"ContainerStarted","Data":"5ec5d746ffe56a8ca2031ca5d5ae7b363515e14d22fd3fdac133270efc27de18"}
Oct 30 23:02:19.513415 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.513389    4581 scope.go:122] "RemoveContainer" containerID="b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988"
Oct 30 23:02:19.513884 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:19.513856    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988\": container with ID starting with b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988 not found: ID does not exist" containerID="b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988"
Oct 30 23:02:19.513955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.513896    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988"} err="failed to get container status \"b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988\": rpc error: code = NotFound desc = could not find container \"b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988\": container with ID starting with b3d859e3781869d8c12bc57d5eb2d373f0f667cf4eccedab6eaa3fb0dbb33988 not found: ID does not exist"
Oct 30 23:02:19.543850 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.543803    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-190f6b" podStartSLOduration=2.543790397 podStartE2EDuration="2.543790397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:19.540494783 +0000 UTC m=+3323.579433671" watchObservedRunningTime="2026-10-30 23:02:19.543790397 +0000 UTC m=+3323.582729283"
Oct 30 23:02:19.576667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.576633    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"]
Oct 30 23:02:19.586890 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:19.586858    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6f2eba-5576597779-g8wwj"]
Oct 30 23:02:20.099957 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:20.099920    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074eaa00-c73a-448d-8c3c-8fe99db910e9" path="/var/lib/kubelet/pods/074eaa00-c73a-448d-8c3c-8fe99db910e9/volumes"
Oct 30 23:02:20.100471 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:20.100451    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e043128-f90a-469a-ae0c-a09a7e9a0654" path="/var/lib/kubelet/pods/3e043128-f90a-469a-ae0c-a09a7e9a0654/volumes"
Oct 30 23:02:20.428409 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:20.428372    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-190f6b"]
Oct 30 23:02:20.433233 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:20.433184    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-190f6b"]
Oct 30 23:02:21.517500 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:21.517461    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-190f6b" podUID="c154ba4b-0e64-4fc3-bc31-8f392e988153" containerName="test-curl-7269-190f6b" containerID="cri-o://73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571" gracePeriod=2
Oct 30 23:02:21.528822 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:21.528786    4581 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-190f6b\" is forbidden: User \"system:node:ip-10-0-135-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-132.ec2.internal' and this object" podUID="c154ba4b-0e64-4fc3-bc31-8f392e988153" pod="opendatahub/test-curl-7269-190f6b"
Oct 30 23:02:23.056598 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.056562    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"]
Oct 30 23:02:23.057240 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.057209    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3e043128-f90a-469a-ae0c-a09a7e9a0654" containerName="payload-processing"
Oct 30 23:02:23.057240 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.057239    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e043128-f90a-469a-ae0c-a09a7e9a0654" containerName="payload-processing"
Oct 30 23:02:23.057402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.057268    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c154ba4b-0e64-4fc3-bc31-8f392e988153" containerName="test-curl-7269-190f6b"
Oct 30 23:02:23.057402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.057275    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c154ba4b-0e64-4fc3-bc31-8f392e988153" containerName="test-curl-7269-190f6b"
Oct 30 23:02:23.057402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.057398    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e043128-f90a-469a-ae0c-a09a7e9a0654" containerName="payload-processing"
Oct 30 23:02:23.057560 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.057409    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c154ba4b-0e64-4fc3-bc31-8f392e988153" containerName="test-curl-7269-190f6b"
Oct 30 23:02:23.062802 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.062777    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.077612 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.077580    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:02:23.077776 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.077676    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:02:23.087531 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.087503    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:02:23.087726 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.087631    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:02:23.095628 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.095595    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"]
Oct 30 23:02:23.181930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.181316    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stlr\" (UniqueName: \"kubernetes.io/projected/f9a61f0c-bcc3-41f8-93cf-91132f43567a-kube-api-access-2stlr\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.181930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.181596    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.181930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.181740    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.283252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.283166    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.283424 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.283322    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.283424 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.283369    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stlr\" (UniqueName: \"kubernetes.io/projected/f9a61f0c-bcc3-41f8-93cf-91132f43567a-kube-api-access-2stlr\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.284041 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.284010    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.286054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.286023    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.300983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.300958    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stlr\" (UniqueName: \"kubernetes.io/projected/f9a61f0c-bcc3-41f8-93cf-91132f43567a-kube-api-access-2stlr\") pod \"maas-api-revoke-sub-03bf421b-qsmck\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.377712 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.377680    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:23.693554 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.693509    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"]
Oct 30 23:02:23.694756 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:02:23.694726    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a61f0c_bcc3_41f8_93cf_91132f43567a.slice/crio-6918a2593e4d83423cdc87560c21b526c20bd1c3a9b12c336bb0ffc509e60414 WatchSource:0}: Error finding container 6918a2593e4d83423cdc87560c21b526c20bd1c3a9b12c336bb0ffc509e60414: Status 404 returned error can't find the container with id 6918a2593e4d83423cdc87560c21b526c20bd1c3a9b12c336bb0ffc509e60414
Oct 30 23:02:23.743742 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.743717    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-190f6b"
Oct 30 23:02:23.791832 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.791800    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c154ba4b-0e64-4fc3-bc31-8f392e988153-kube-api-access-w7wkd\" (UniqueName: \"kubernetes.io/projected/c154ba4b-0e64-4fc3-bc31-8f392e988153-kube-api-access-w7wkd\") pod \"c154ba4b-0e64-4fc3-bc31-8f392e988153\" (UID: \"c154ba4b-0e64-4fc3-bc31-8f392e988153\") "
Oct 30 23:02:23.851988 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.851951    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c154ba4b-0e64-4fc3-bc31-8f392e988153-kube-api-access-w7wkd" pod "c154ba4b-0e64-4fc3-bc31-8f392e988153" (UID: "c154ba4b-0e64-4fc3-bc31-8f392e988153"). InnerVolumeSpecName "kube-api-access-w7wkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:23.894172 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:23.894137    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wkd\" (UniqueName: \"kubernetes.io/projected/c154ba4b-0e64-4fc3-bc31-8f392e988153-kube-api-access-w7wkd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:24.099201 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.099166    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c154ba4b-0e64-4fc3-bc31-8f392e988153" path="/var/lib/kubelet/pods/c154ba4b-0e64-4fc3-bc31-8f392e988153/volumes"
Oct 30 23:02:24.546771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.546733    4581 generic.go:356] "Generic (PLEG): container finished" podID="c154ba4b-0e64-4fc3-bc31-8f392e988153" containerID="73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571" exitCode=137
Oct 30 23:02:24.546975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.546778    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-190f6b"
Oct 30 23:02:24.546975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.546820    4581 scope.go:122] "RemoveContainer" containerID="73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571"
Oct 30 23:02:24.549310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.549019    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck" event={"ID":"f9a61f0c-bcc3-41f8-93cf-91132f43567a","Type":"ContainerStarted","Data":"21a2e264da0ce4f2b29bbbdbe093ca25cc1a89ad1cb70ca1925d7f3dcf382911"}
Oct 30 23:02:24.549310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.549078    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck" event={"ID":"f9a61f0c-bcc3-41f8-93cf-91132f43567a","Type":"ContainerStarted","Data":"6918a2593e4d83423cdc87560c21b526c20bd1c3a9b12c336bb0ffc509e60414"}
Oct 30 23:02:24.562088 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.562054    4581 scope.go:122] "RemoveContainer" containerID="73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571"
Oct 30 23:02:24.562475 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:24.562444    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571\": container with ID starting with 73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571 not found: ID does not exist" containerID="73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571"
Oct 30 23:02:24.562567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.562485    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571"} err="failed to get container status \"73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571\": rpc error: code = NotFound desc = could not find container \"73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571\": container with ID starting with 73782d4b5686ef85ca35283f2648d8d0b4f308bf834f6e7088b9af48d1b83571 not found: ID does not exist"
Oct 30 23:02:24.590624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:24.590568    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck" podStartSLOduration=1.590549499 podStartE2EDuration="1.590549499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:24.585045312 +0000 UTC m=+3328.623984200" watchObservedRunningTime="2026-10-30 23:02:24.590549499 +0000 UTC m=+3328.629488386"
Oct 30 23:02:24.607008 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:24.606982    4581 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a61f0c_bcc3_41f8_93cf_91132f43567a.slice/crio-21a2e264da0ce4f2b29bbbdbe093ca25cc1a89ad1cb70ca1925d7f3dcf382911.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 23:02:25.593880 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:25.593473    4581 generic.go:356] "Generic (PLEG): container finished" podID="f9a61f0c-bcc3-41f8-93cf-91132f43567a" containerID="21a2e264da0ce4f2b29bbbdbe093ca25cc1a89ad1cb70ca1925d7f3dcf382911" exitCode=0
Oct 30 23:02:25.593880 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:25.593612    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck" event={"ID":"f9a61f0c-bcc3-41f8-93cf-91132f43567a","Type":"ContainerDied","Data":"21a2e264da0ce4f2b29bbbdbe093ca25cc1a89ad1cb70ca1925d7f3dcf382911"}
Oct 30 23:02:26.164624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.164583    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"]
Oct 30 23:02:26.170609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.170584    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.191713 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.191683    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"]
Oct 30 23:02:26.233746 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.233709    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.234080 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.234053    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzftt\" (UniqueName: \"kubernetes.io/projected/b7fd0369-9802-468e-80d2-0dde78473b51-kube-api-access-tzftt\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.234193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.234111    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.336072 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.336032    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzftt\" (UniqueName: \"kubernetes.io/projected/b7fd0369-9802-468e-80d2-0dde78473b51-kube-api-access-tzftt\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.336324 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.336086    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.336324 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.336133    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.336851 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.336825    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.338779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.338758    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.354282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.354259    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzftt\" (UniqueName: \"kubernetes.io/projected/b7fd0369-9802-468e-80d2-0dde78473b51-kube-api-access-tzftt\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.488619 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.488527    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:26.802335 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.802298    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"]
Oct 30 23:02:26.803993 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:02:26.803963    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fd0369_9802_468e_80d2_0dde78473b51.slice/crio-9da7c9c7f68c1fddbf3215b445e902579d02649465965bf5fb173e04985a078b WatchSource:0}: Error finding container 9da7c9c7f68c1fddbf3215b445e902579d02649465965bf5fb173e04985a078b: Status 404 returned error can't find the container with id 9da7c9c7f68c1fddbf3215b445e902579d02649465965bf5fb173e04985a078b
Oct 30 23:02:26.825781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.825754    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:26.966241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.958939    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a61f0c-bcc3-41f8-93cf-91132f43567a-kube-api-access-2stlr\" (UniqueName: \"kubernetes.io/projected/f9a61f0c-bcc3-41f8-93cf-91132f43567a-kube-api-access-2stlr\") pod \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") "
Oct 30 23:02:26.966241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.959055    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-openshift-ca\") pod \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") "
Oct 30 23:02:26.966241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.959174    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-xks-ca\") pod \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\" (UID: \"f9a61f0c-bcc3-41f8-93cf-91132f43567a\") "
Oct 30 23:02:26.966241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.962625    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-xks-ca" pod "f9a61f0c-bcc3-41f8-93cf-91132f43567a" (UID: "f9a61f0c-bcc3-41f8-93cf-91132f43567a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:26.966241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.965063    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a61f0c-bcc3-41f8-93cf-91132f43567a-kube-api-access-2stlr" pod "f9a61f0c-bcc3-41f8-93cf-91132f43567a" (UID: "f9a61f0c-bcc3-41f8-93cf-91132f43567a"). InnerVolumeSpecName "kube-api-access-2stlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:26.966241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:26.965430    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-openshift-ca" pod "f9a61f0c-bcc3-41f8-93cf-91132f43567a" (UID: "f9a61f0c-bcc3-41f8-93cf-91132f43567a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:27.061463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.061376    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stlr\" (UniqueName: \"kubernetes.io/projected/f9a61f0c-bcc3-41f8-93cf-91132f43567a-kube-api-access-2stlr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:27.061463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.061410    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:27.061463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.061428    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a61f0c-bcc3-41f8-93cf-91132f43567a-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:27.617654 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.617626    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"
Oct 30 23:02:27.617837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.617659    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck" event={"ID":"f9a61f0c-bcc3-41f8-93cf-91132f43567a","Type":"ContainerDied","Data":"6918a2593e4d83423cdc87560c21b526c20bd1c3a9b12c336bb0ffc509e60414"}
Oct 30 23:02:27.617837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.617713    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6918a2593e4d83423cdc87560c21b526c20bd1c3a9b12c336bb0ffc509e60414"
Oct 30 23:02:27.620078 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.620049    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v" event={"ID":"b7fd0369-9802-468e-80d2-0dde78473b51","Type":"ContainerStarted","Data":"348b0ab9354fa68bfa1e3a00f89426023542e5cd414c5c0b4e8fd4bc9c722f16"}
Oct 30 23:02:27.620266 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.620105    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v" event={"ID":"b7fd0369-9802-468e-80d2-0dde78473b51","Type":"ContainerStarted","Data":"9da7c9c7f68c1fddbf3215b445e902579d02649465965bf5fb173e04985a078b"}
Oct 30 23:02:27.662449 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:27.662386    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v" podStartSLOduration=1.662368499 podStartE2EDuration="1.662368499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:27.656648205 +0000 UTC m=+3331.695587092" watchObservedRunningTime="2026-10-30 23:02:27.662368499 +0000 UTC m=+3331.701307386"
Oct 30 23:02:28.630955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:28.630923    4581 generic.go:356] "Generic (PLEG): container finished" podID="b7fd0369-9802-468e-80d2-0dde78473b51" containerID="348b0ab9354fa68bfa1e3a00f89426023542e5cd414c5c0b4e8fd4bc9c722f16" exitCode=0
Oct 30 23:02:28.631509 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:28.630998    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v" event={"ID":"b7fd0369-9802-468e-80d2-0dde78473b51","Type":"ContainerDied","Data":"348b0ab9354fa68bfa1e3a00f89426023542e5cd414c5c0b4e8fd4bc9c722f16"}
Oct 30 23:02:29.851626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:29.851595    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:29.910717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:29.910630    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-xks-ca\") pod \"b7fd0369-9802-468e-80d2-0dde78473b51\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") "
Oct 30 23:02:29.910911 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:29.910807    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-openshift-ca\") pod \"b7fd0369-9802-468e-80d2-0dde78473b51\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") "
Oct 30 23:02:29.911033 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:29.911007    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fd0369-9802-468e-80d2-0dde78473b51-kube-api-access-tzftt\" (UniqueName: \"kubernetes.io/projected/b7fd0369-9802-468e-80d2-0dde78473b51-kube-api-access-tzftt\") pod \"b7fd0369-9802-468e-80d2-0dde78473b51\" (UID: \"b7fd0369-9802-468e-80d2-0dde78473b51\") "
Oct 30 23:02:29.911449 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:29.911415    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-openshift-ca" pod "b7fd0369-9802-468e-80d2-0dde78473b51" (UID: "b7fd0369-9802-468e-80d2-0dde78473b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:29.912567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:29.912541    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:29.914773 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:29.914330    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fd0369-9802-468e-80d2-0dde78473b51-kube-api-access-tzftt" pod "b7fd0369-9802-468e-80d2-0dde78473b51" (UID: "b7fd0369-9802-468e-80d2-0dde78473b51"). InnerVolumeSpecName "kube-api-access-tzftt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:29.915147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:29.915121    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-xks-ca" pod "b7fd0369-9802-468e-80d2-0dde78473b51" (UID: "b7fd0369-9802-468e-80d2-0dde78473b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:30.014042 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:30.014008    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzftt\" (UniqueName: \"kubernetes.io/projected/b7fd0369-9802-468e-80d2-0dde78473b51-kube-api-access-tzftt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.014042 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:30.014041    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fd0369-9802-468e-80d2-0dde78473b51-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:30.660257 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:30.660229    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"
Oct 30 23:02:30.660463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:30.660283    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v" event={"ID":"b7fd0369-9802-468e-80d2-0dde78473b51","Type":"ContainerDied","Data":"9da7c9c7f68c1fddbf3215b445e902579d02649465965bf5fb173e04985a078b"}
Oct 30 23:02:30.660463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:30.660338    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da7c9c7f68c1fddbf3215b445e902579d02649465965bf5fb173e04985a078b"
Oct 30 23:02:30.721205 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:30.721162    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"]
Oct 30 23:02:30.727515 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:30.727482    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0a807-78xzm"]
Oct 30 23:02:32.070508 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:32.070476    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"]
Oct 30 23:02:32.080329 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:32.080302    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0eb37a-qvswf"]
Oct 30 23:02:32.099758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:32.099727    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5" path="/var/lib/kubelet/pods/d2cfb32f-3e3d-47c4-b5b3-0509e8a3f0e5/volumes"
Oct 30 23:02:32.100205 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:32.100190    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07671ac-eade-4722-8783-afa3a5ac8b21" path="/var/lib/kubelet/pods/f07671ac-eade-4722-8783-afa3a5ac8b21/volumes"
Oct 30 23:02:33.405639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.405609    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"]
Oct 30 23:02:33.406312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.406296    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b7fd0369-9802-468e-80d2-0dde78473b51" containerName="revoke-keys"
Oct 30 23:02:33.406312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.406312    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fd0369-9802-468e-80d2-0dde78473b51" containerName="revoke-keys"
Oct 30 23:02:33.406428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.406328    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f9a61f0c-bcc3-41f8-93cf-91132f43567a" containerName="revoke-keys"
Oct 30 23:02:33.406428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.406334    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a61f0c-bcc3-41f8-93cf-91132f43567a" containerName="revoke-keys"
Oct 30 23:02:33.406496 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.406454    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fd0369-9802-468e-80d2-0dde78473b51" containerName="revoke-keys"
Oct 30 23:02:33.406496 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.406466    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a61f0c-bcc3-41f8-93cf-91132f43567a" containerName="revoke-keys"
Oct 30 23:02:33.411435 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.411413    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.429672 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.429647    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"]
Oct 30 23:02:33.434145 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.434123    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:02:33.434314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.434129    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:02:33.434314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.434153    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:02:33.434314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.434278    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:02:33.569301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.569261    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2khh\" (UniqueName: \"kubernetes.io/projected/ad2654ae-cd6d-4714-9514-86cf17f9ef31-kube-api-access-n2khh\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.569519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.569386    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.569519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.569513    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.671491 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.671405    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.671667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.671521    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.671727 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.671696    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2khh\" (UniqueName: \"kubernetes.io/projected/ad2654ae-cd6d-4714-9514-86cf17f9ef31-kube-api-access-n2khh\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.672289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.672265    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.674151 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.674125    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.690392 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.690365    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2khh\" (UniqueName: \"kubernetes.io/projected/ad2654ae-cd6d-4714-9514-86cf17f9ef31-kube-api-access-n2khh\") pod \"maas-api-revoke-sub-1284f0d0-7sntv\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:33.726330 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:33.726300    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:34.040755 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:34.040725    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"]
Oct 30 23:02:34.042596 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:02:34.042567    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2654ae_cd6d_4714_9514_86cf17f9ef31.slice/crio-081dde46c7862ba817654edb92423a0c8fb2bbfc0faa25318a0e709987605fa1 WatchSource:0}: Error finding container 081dde46c7862ba817654edb92423a0c8fb2bbfc0faa25318a0e709987605fa1: Status 404 returned error can't find the container with id 081dde46c7862ba817654edb92423a0c8fb2bbfc0faa25318a0e709987605fa1
Oct 30 23:02:34.895834 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:34.895793    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv" event={"ID":"ad2654ae-cd6d-4714-9514-86cf17f9ef31","Type":"ContainerStarted","Data":"aeed745158c006c1e978acf03a7437812c65aa010a6514466419f434a826758a"}
Oct 30 23:02:34.896282 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:34.895874    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv" event={"ID":"ad2654ae-cd6d-4714-9514-86cf17f9ef31","Type":"ContainerStarted","Data":"081dde46c7862ba817654edb92423a0c8fb2bbfc0faa25318a0e709987605fa1"}
Oct 30 23:02:34.931364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:34.931257    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv" podStartSLOduration=1.931241583 podStartE2EDuration="1.931241583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:34.929588929 +0000 UTC m=+3338.968527811" watchObservedRunningTime="2026-10-30 23:02:34.931241583 +0000 UTC m=+3338.970180469"
Oct 30 23:02:34.982128 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:34.982091    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"]
Oct 30 23:02:34.986611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:34.986582    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6f2eba-cjdvk"]
Oct 30 23:02:35.906980 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:35.906950    4581 generic.go:356] "Generic (PLEG): container finished" podID="ad2654ae-cd6d-4714-9514-86cf17f9ef31" containerID="aeed745158c006c1e978acf03a7437812c65aa010a6514466419f434a826758a" exitCode=0
Oct 30 23:02:35.907423 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:35.907038    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv" event={"ID":"ad2654ae-cd6d-4714-9514-86cf17f9ef31","Type":"ContainerDied","Data":"aeed745158c006c1e978acf03a7437812c65aa010a6514466419f434a826758a"}
Oct 30 23:02:36.099788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:36.099753    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd2cd81-751f-4b6f-a51e-fa109644f7f4" path="/var/lib/kubelet/pods/9dd2cd81-751f-4b6f-a51e-fa109644f7f4/volumes"
Oct 30 23:02:37.084429 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.084401    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:37.174779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.174691    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-openshift-ca\") pod \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") "
Oct 30 23:02:37.174931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.174869    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2654ae-cd6d-4714-9514-86cf17f9ef31-kube-api-access-n2khh\" (UniqueName: \"kubernetes.io/projected/ad2654ae-cd6d-4714-9514-86cf17f9ef31-kube-api-access-n2khh\") pod \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") "
Oct 30 23:02:37.174931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.174903    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-xks-ca\") pod \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\" (UID: \"ad2654ae-cd6d-4714-9514-86cf17f9ef31\") "
Oct 30 23:02:37.175090 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.175063    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-openshift-ca" pod "ad2654ae-cd6d-4714-9514-86cf17f9ef31" (UID: "ad2654ae-cd6d-4714-9514-86cf17f9ef31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:37.175844 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.175817    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:37.177507 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.177478    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-xks-ca" pod "ad2654ae-cd6d-4714-9514-86cf17f9ef31" (UID: "ad2654ae-cd6d-4714-9514-86cf17f9ef31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:37.177646 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.177537    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2654ae-cd6d-4714-9514-86cf17f9ef31-kube-api-access-n2khh" pod "ad2654ae-cd6d-4714-9514-86cf17f9ef31" (UID: "ad2654ae-cd6d-4714-9514-86cf17f9ef31"). InnerVolumeSpecName "kube-api-access-n2khh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:37.283246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.282516    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2khh\" (UniqueName: \"kubernetes.io/projected/ad2654ae-cd6d-4714-9514-86cf17f9ef31-kube-api-access-n2khh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:37.283246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.282556    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2654ae-cd6d-4714-9514-86cf17f9ef31-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:37.928398 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.928371    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"
Oct 30 23:02:37.928580 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.928407    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv" event={"ID":"ad2654ae-cd6d-4714-9514-86cf17f9ef31","Type":"ContainerDied","Data":"081dde46c7862ba817654edb92423a0c8fb2bbfc0faa25318a0e709987605fa1"}
Oct 30 23:02:37.928580 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:37.928452    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="081dde46c7862ba817654edb92423a0c8fb2bbfc0faa25318a0e709987605fa1"
Oct 30 23:02:38.358277 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.358235    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"]
Oct 30 23:02:38.358785 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.358499    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy" containerID="cri-o://d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e" gracePeriod=30
Oct 30 23:02:38.896184 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.896058    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"]
Oct 30 23:02:38.897762 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.897727    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ad2654ae-cd6d-4714-9514-86cf17f9ef31" containerName="revoke-keys"
Oct 30 23:02:38.897762 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.897762    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2654ae-cd6d-4714-9514-86cf17f9ef31" containerName="revoke-keys"
Oct 30 23:02:38.898061 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.898045    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2654ae-cd6d-4714-9514-86cf17f9ef31" containerName="revoke-keys"
Oct 30 23:02:38.946050 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.945588    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"]
Oct 30 23:02:38.946050 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.945712    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:38.958658 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.958632    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:02:38.958960 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.958942    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:02:38.961089 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.961060    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:02:38.961236 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:38.961147    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:02:39.025139 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.025100    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwjp8\" (UniqueName: \"kubernetes.io/projected/a2582fac-e117-49ec-8399-5c3a5a381773-kube-api-access-zwjp8\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.025358 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.025288    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.025440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.025423    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.127416 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.127384    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwjp8\" (UniqueName: \"kubernetes.io/projected/a2582fac-e117-49ec-8399-5c3a5a381773-kube-api-access-zwjp8\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.127702 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.127499    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.127702 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.127615    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.128194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.128169    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.130359 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.130334    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.145656 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.145626    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwjp8\" (UniqueName: \"kubernetes.io/projected/a2582fac-e117-49ec-8399-5c3a5a381773-kube-api-access-zwjp8\") pod \"maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.266876 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.266775    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:39.579606 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.579570    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"]
Oct 30 23:02:39.580106 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:02:39.580081    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2582fac_e117_49ec_8399_5c3a5a381773.slice/crio-a7be22dc90522876a94a87efbb09137dcf0dabe327d7ed80e3c46e1ab77048d1 WatchSource:0}: Error finding container a7be22dc90522876a94a87efbb09137dcf0dabe327d7ed80e3c46e1ab77048d1: Status 404 returned error can't find the container with id a7be22dc90522876a94a87efbb09137dcf0dabe327d7ed80e3c46e1ab77048d1
Oct 30 23:02:39.968020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:39.967991    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk" event={"ID":"a2582fac-e117-49ec-8399-5c3a5a381773","Type":"ContainerStarted","Data":"a7be22dc90522876a94a87efbb09137dcf0dabe327d7ed80e3c46e1ab77048d1"}
Oct 30 23:02:40.592259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:40.592208    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"]
Oct 30 23:02:40.592608 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:40.592472    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j" podUID="550908fa-315e-4014-88c1-075b25a56178" containerName="payload-processing" containerID="cri-o://7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4" gracePeriod=30
Oct 30 23:02:40.609433 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:40.609398    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"]
Oct 30 23:02:40.861794 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:40.859161    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:02:40.908962 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:40.908935    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 23:02:41.042243 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.018860    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550908fa-315e-4014-88c1-075b25a56178-kube-api-access-rl5c7\" (UniqueName: \"kubernetes.io/projected/550908fa-315e-4014-88c1-075b25a56178-kube-api-access-rl5c7\") pod \"550908fa-315e-4014-88c1-075b25a56178\" (UID: \"550908fa-315e-4014-88c1-075b25a56178\") "
Oct 30 23:02:41.101629 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.101589    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550908fa-315e-4014-88c1-075b25a56178-kube-api-access-rl5c7" pod "550908fa-315e-4014-88c1-075b25a56178" (UID: "550908fa-315e-4014-88c1-075b25a56178"). InnerVolumeSpecName "kube-api-access-rl5c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:41.111700 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.111661    4581 patch_prober.go:29] interesting pod/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:02:41.111917 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.111725    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:02:41.127579 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.125588    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550908fa-315e-4014-88c1-075b25a56178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550908fa-315e-4014-88c1-075b25a56178-plugins-config-volume\") pod \"550908fa-315e-4014-88c1-075b25a56178\" (UID: \"550908fa-315e-4014-88c1-075b25a56178\") "
Oct 30 23:02:41.127579 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.126139    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550908fa-315e-4014-88c1-075b25a56178-plugins-config-volume" pod "550908fa-315e-4014-88c1-075b25a56178" (UID: "550908fa-315e-4014-88c1-075b25a56178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:41.127579 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.126678    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550908fa-315e-4014-88c1-075b25a56178-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:41.127579 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.126948    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5c7\" (UniqueName: \"kubernetes.io/projected/550908fa-315e-4014-88c1-075b25a56178-kube-api-access-rl5c7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:41.129631 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.129600    4581 generic.go:356] "Generic (PLEG): container finished" podID="550908fa-315e-4014-88c1-075b25a56178" containerID="7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4" exitCode=0
Oct 30 23:02:41.129781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.129698    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j" event={"ID":"550908fa-315e-4014-88c1-075b25a56178","Type":"ContainerDied","Data":"7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4"}
Oct 30 23:02:41.129781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.129745    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j" event={"ID":"550908fa-315e-4014-88c1-075b25a56178","Type":"ContainerDied","Data":"73bd64bb3b5240983b459488f252e6a7b2513b78cb1e59374757db7a39a6846b"}
Oct 30 23:02:41.129781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.129771    4581 scope.go:122] "RemoveContainer" containerID="7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4"
Oct 30 23:02:41.131634 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.131581    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"
Oct 30 23:02:41.140184 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.140148    4581 generic.go:356] "Generic (PLEG): container finished" podID="a2582fac-e117-49ec-8399-5c3a5a381773" containerID="51aa94efb9f79980efa32b2001ad45f6c61958a44cd7f19171bdbd051d33e8a4" exitCode=0
Oct 30 23:02:41.140420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.140394    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk" event={"ID":"a2582fac-e117-49ec-8399-5c3a5a381773","Type":"ContainerDied","Data":"51aa94efb9f79980efa32b2001ad45f6c61958a44cd7f19171bdbd051d33e8a4"}
Oct 30 23:02:41.140578 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.140475    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp" podUID="b177e235-e812-4385-aada-80fc0e4874c4" containerName="payload-pre-processing" containerID="cri-o://954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2" gracePeriod=30
Oct 30 23:02:41.160502 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.160473    4581 scope.go:122] "RemoveContainer" containerID="7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4"
Oct 30 23:02:41.160866 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:41.160833    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4\": container with ID starting with 7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4 not found: ID does not exist" containerID="7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4"
Oct 30 23:02:41.160951 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.160880    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4"} err="failed to get container status \"7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4\": rpc error: code = NotFound desc = could not find container \"7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4\": container with ID starting with 7bf80687b5ac9c57b9cc74e9aac8072c34e9820b1d13bea6511321f8839eeda4 not found: ID does not exist"
Oct 30 23:02:41.262392 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.262362    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"]
Oct 30 23:02:41.267894 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.267866    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88979-74fd6f8bd-vfz9j"]
Oct 30 23:02:41.314972 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.314327    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"]
Oct 30 23:02:41.317352 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.317320    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f88979-vzzpm"]
Oct 30 23:02:41.470162 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.470131    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 23:02:41.532534 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.532478    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b177e235-e812-4385-aada-80fc0e4874c4-kube-api-access-2d7qt\" (UniqueName: \"kubernetes.io/projected/b177e235-e812-4385-aada-80fc0e4874c4-kube-api-access-2d7qt\") pod \"b177e235-e812-4385-aada-80fc0e4874c4\" (UID: \"b177e235-e812-4385-aada-80fc0e4874c4\") "
Oct 30 23:02:41.532534 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.532536    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b177e235-e812-4385-aada-80fc0e4874c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b177e235-e812-4385-aada-80fc0e4874c4-plugins-config-volume\") pod \"b177e235-e812-4385-aada-80fc0e4874c4\" (UID: \"b177e235-e812-4385-aada-80fc0e4874c4\") "
Oct 30 23:02:41.533036 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.533004    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b177e235-e812-4385-aada-80fc0e4874c4-plugins-config-volume" pod "b177e235-e812-4385-aada-80fc0e4874c4" (UID: "b177e235-e812-4385-aada-80fc0e4874c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:41.533243 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.533152    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b177e235-e812-4385-aada-80fc0e4874c4-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:41.535072 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.535048    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b177e235-e812-4385-aada-80fc0e4874c4-kube-api-access-2d7qt" pod "b177e235-e812-4385-aada-80fc0e4874c4" (UID: "b177e235-e812-4385-aada-80fc0e4874c4"). InnerVolumeSpecName "kube-api-access-2d7qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:41.634105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:41.634065    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d7qt\" (UniqueName: \"kubernetes.io/projected/b177e235-e812-4385-aada-80fc0e4874c4-kube-api-access-2d7qt\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:42.099933 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.099899    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550908fa-315e-4014-88c1-075b25a56178" path="/var/lib/kubelet/pods/550908fa-315e-4014-88c1-075b25a56178/volumes"
Oct 30 23:02:42.100436 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.100420    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b95030-87ba-495f-ba9b-9c4c9ac4af5a" path="/var/lib/kubelet/pods/89b95030-87ba-495f-ba9b-9c4c9ac4af5a/volumes"
Oct 30 23:02:42.153005 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.152933    4581 generic.go:356] "Generic (PLEG): container finished" podID="b177e235-e812-4385-aada-80fc0e4874c4" containerID="954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2" exitCode=0
Oct 30 23:02:42.153005 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.153005    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"
Oct 30 23:02:42.153468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.153434    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp" event={"ID":"b177e235-e812-4385-aada-80fc0e4874c4","Type":"ContainerDied","Data":"954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2"}
Oct 30 23:02:42.153563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.153494    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp" event={"ID":"b177e235-e812-4385-aada-80fc0e4874c4","Type":"ContainerDied","Data":"eea90e6b8dfbcfd9d90657ff1fec669fdbd81d6edd0df5d3238c345f1c09b209"}
Oct 30 23:02:42.153563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.153529    4581 scope.go:122] "RemoveContainer" containerID="954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2"
Oct 30 23:02:42.169197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.169170    4581 scope.go:122] "RemoveContainer" containerID="954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2"
Oct 30 23:02:42.169602 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:42.169569    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2\": container with ID starting with 954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2 not found: ID does not exist" containerID="954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2"
Oct 30 23:02:42.169719 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.169614    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2"} err="failed to get container status \"954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2\": rpc error: code = NotFound desc = could not find container \"954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2\": container with ID starting with 954e134ac332bc8d0b5bdeaaf8f856f9928e48301961403341572f056c438fa2 not found: ID does not exist"
Oct 30 23:02:42.201497 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.201457    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"]
Oct 30 23:02:42.211888 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.211855    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88979-669bd98b7f-9lsrp"]
Oct 30 23:02:42.343286 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.343210    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:42.345820 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.345792    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2582fac-e117-49ec-8399-5c3a5a381773-kube-api-access-zwjp8\" (UniqueName: \"kubernetes.io/projected/a2582fac-e117-49ec-8399-5c3a5a381773-kube-api-access-zwjp8\") pod \"a2582fac-e117-49ec-8399-5c3a5a381773\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") "
Oct 30 23:02:42.346047 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.346029    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-openshift-ca\") pod \"a2582fac-e117-49ec-8399-5c3a5a381773\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") "
Oct 30 23:02:42.346135 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.346071    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-xks-ca\") pod \"a2582fac-e117-49ec-8399-5c3a5a381773\" (UID: \"a2582fac-e117-49ec-8399-5c3a5a381773\") "
Oct 30 23:02:42.346442 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.346417    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-openshift-ca" pod "a2582fac-e117-49ec-8399-5c3a5a381773" (UID: "a2582fac-e117-49ec-8399-5c3a5a381773"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:42.346617 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.346594    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:42.348301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.348277    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-xks-ca" pod "a2582fac-e117-49ec-8399-5c3a5a381773" (UID: "a2582fac-e117-49ec-8399-5c3a5a381773"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:42.348706 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.348692    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2582fac-e117-49ec-8399-5c3a5a381773-kube-api-access-zwjp8" pod "a2582fac-e117-49ec-8399-5c3a5a381773" (UID: "a2582fac-e117-49ec-8399-5c3a5a381773"). InnerVolumeSpecName "kube-api-access-zwjp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:42.447876 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.447835    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwjp8\" (UniqueName: \"kubernetes.io/projected/a2582fac-e117-49ec-8399-5c3a5a381773-kube-api-access-zwjp8\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:42.447876 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.447869    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2582fac-e117-49ec-8399-5c3a5a381773-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:42.593533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.593499    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"]
Oct 30 23:02:42.593742 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:42.593715    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy" containerID="cri-o://6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028" gracePeriod=30
Oct 30 23:02:43.032606 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.032566    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"]
Oct 30 23:02:43.037792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.037757    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79790862-7b9ms"]
Oct 30 23:02:43.174088 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.174059    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"
Oct 30 23:02:43.174306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.174055    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk" event={"ID":"a2582fac-e117-49ec-8399-5c3a5a381773","Type":"ContainerDied","Data":"a7be22dc90522876a94a87efbb09137dcf0dabe327d7ed80e3c46e1ab77048d1"}
Oct 30 23:02:43.174306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.174184    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7be22dc90522876a94a87efbb09137dcf0dabe327d7ed80e3c46e1ab77048d1"
Oct 30 23:02:43.198744 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.198710    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"]
Oct 30 23:02:43.199758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.199738    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b177e235-e812-4385-aada-80fc0e4874c4" containerName="payload-pre-processing"
Oct 30 23:02:43.199826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.199762    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b177e235-e812-4385-aada-80fc0e4874c4" containerName="payload-pre-processing"
Oct 30 23:02:43.199826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.199801    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a2582fac-e117-49ec-8399-5c3a5a381773" containerName="revoke-keys"
Oct 30 23:02:43.199826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.199809    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2582fac-e117-49ec-8399-5c3a5a381773" containerName="revoke-keys"
Oct 30 23:02:43.199930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.199826    4581 cpu_manager.go:414] "containerMap: removing container" podUID="550908fa-315e-4014-88c1-075b25a56178" containerName="payload-processing"
Oct 30 23:02:43.199930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.199835    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550908fa-315e-4014-88c1-075b25a56178" containerName="payload-processing"
Oct 30 23:02:43.200016 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.200004    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b177e235-e812-4385-aada-80fc0e4874c4" containerName="payload-pre-processing"
Oct 30 23:02:43.200052 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.200023    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2582fac-e117-49ec-8399-5c3a5a381773" containerName="revoke-keys"
Oct 30 23:02:43.200052 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.200034    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="550908fa-315e-4014-88c1-075b25a56178" containerName="payload-processing"
Oct 30 23:02:43.204568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.204548    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.219633 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.219603    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:02:43.219843 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.219660    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:02:43.221326 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.221303    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"]
Oct 30 23:02:43.259123 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.259085    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.259331 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.259286    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfp8b\" (UniqueName: \"kubernetes.io/projected/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-kube-api-access-tfp8b\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.259530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.259507    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.291606 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.291492    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:02:43.291762 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.291610    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:02:43.360845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.360807    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfp8b\" (UniqueName: \"kubernetes.io/projected/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-kube-api-access-tfp8b\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.361076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.360919    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.361076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.360990    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.361791 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.361710    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.363686 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.363661    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.380562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.380534    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfp8b\" (UniqueName: \"kubernetes.io/projected/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-kube-api-access-tfp8b\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.521428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.521393    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:43.685380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.685152    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 23:02:43.769828 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.769787    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-certs\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.770051 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.769852    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-envoy\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.770051 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.769920    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-kube-api-access-jfjcn\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-kube-api-access-jfjcn\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.770174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.770046    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-podinfo\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.770174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.770088    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-credential-socket\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.770337 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.770176    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-certs" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:43.770337 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.770308    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-data\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.770463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.770343    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd29e823-643c-4ef9-b94f-dbffd2461749-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd29e823-643c-4ef9-b94f-dbffd2461749-istiod-ca-cert\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.770463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.770372    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-socket\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.770463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.770424    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-token\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-token\") pod \"dd29e823-643c-4ef9-b94f-dbffd2461749\" (UID: \"dd29e823-643c-4ef9-b94f-dbffd2461749\") "
Oct 30 23:02:43.771075 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.771041    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd29e823-643c-4ef9-b94f-dbffd2461749-istiod-ca-cert" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:43.771209 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.771161    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd29e823-643c-4ef9-b94f-dbffd2461749-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.771209 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.771182    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.771476 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.771452    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-socket" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:43.772446 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.772417    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-data" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:43.772611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.772587    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-credential-socket" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:43.779419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.779368    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-token" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:43.782714 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.782676    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-envoy" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:43.782856 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.782793    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-kube-api-access-jfjcn" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "kube-api-access-jfjcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:43.786395 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.786354    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-podinfo" pod "dd29e823-643c-4ef9-b94f-dbffd2461749" (UID: "dd29e823-643c-4ef9-b94f-dbffd2461749"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:02:43.869877 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.869843    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"]
Oct 30 23:02:43.873162 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.873134    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.873162 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.873165    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.873353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.873178    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.873353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.873191    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.873353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.873207    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfjcn\" (UniqueName: \"kubernetes.io/projected/dd29e823-643c-4ef9-b94f-dbffd2461749-kube-api-access-jfjcn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.873353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.873236    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd29e823-643c-4ef9-b94f-dbffd2461749-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.873353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:43.873250    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd29e823-643c-4ef9-b94f-dbffd2461749-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:43.873557 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:02:43.873462    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4563a2d_4edd_40e7_a786_3fdcde95e0f5.slice/crio-4d53124bee2592dd278149224eb2b6c91e58f67ec5012fca1bef486694e2bc06 WatchSource:0}: Error finding container 4d53124bee2592dd278149224eb2b6c91e58f67ec5012fca1bef486694e2bc06: Status 404 returned error can't find the container with id 4d53124bee2592dd278149224eb2b6c91e58f67ec5012fca1bef486694e2bc06
Oct 30 23:02:44.100499 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.100465    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09485f12-6602-4bff-a5b3-1dc43a9309c0" path="/var/lib/kubelet/pods/09485f12-6602-4bff-a5b3-1dc43a9309c0/volumes"
Oct 30 23:02:44.100919 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.100905    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b177e235-e812-4385-aada-80fc0e4874c4" path="/var/lib/kubelet/pods/b177e235-e812-4385-aada-80fc0e4874c4/volumes"
Oct 30 23:02:44.189726 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.189694    4581 generic.go:356] "Generic (PLEG): container finished" podID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerID="d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e" exitCode=0
Oct 30 23:02:44.189927 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.189771    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"
Oct 30 23:02:44.189927 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.189808    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" event={"ID":"dd29e823-643c-4ef9-b94f-dbffd2461749","Type":"ContainerDied","Data":"d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e"}
Oct 30 23:02:44.189927 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.189877    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr" event={"ID":"dd29e823-643c-4ef9-b94f-dbffd2461749","Type":"ContainerDied","Data":"d46665546ef2b1a658fa7c36a63e92b0d4448b6df67ff47a2732586ab4664d52"}
Oct 30 23:02:44.189927 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.189909    4581 scope.go:122] "RemoveContainer" containerID="d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e"
Oct 30 23:02:44.192188 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.192159    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2" event={"ID":"d4563a2d-4edd-40e7-a786-3fdcde95e0f5","Type":"ContainerStarted","Data":"4d53124bee2592dd278149224eb2b6c91e58f67ec5012fca1bef486694e2bc06"}
Oct 30 23:02:44.206583 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.206561    4581 scope.go:122] "RemoveContainer" containerID="1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2"
Oct 30 23:02:44.218549 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.218529    4581 scope.go:122] "RemoveContainer" containerID="d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e"
Oct 30 23:02:44.218937 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:44.218916    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e\": container with ID starting with d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e not found: ID does not exist" containerID="d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e"
Oct 30 23:02:44.218997 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.218947    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e"} err="failed to get container status \"d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e\": rpc error: code = NotFound desc = could not find container \"d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e\": container with ID starting with d1a095593a5f64b833cc65c04e5a287c86162df769eade33c9ff8522a6d70d4e not found: ID does not exist"
Oct 30 23:02:44.218997 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.218966    4581 scope.go:122] "RemoveContainer" containerID="1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2"
Oct 30 23:02:44.219327 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:44.219308    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2\": container with ID starting with 1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2 not found: ID does not exist" containerID="1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2"
Oct 30 23:02:44.219388 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.219335    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2"} err="failed to get container status \"1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2\": rpc error: code = NotFound desc = could not find container \"1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2\": container with ID starting with 1c9cb061956f9c49e6d1efb7a7d80dcf48731aa48b5fd6b1a45266c9605446c2 not found: ID does not exist"
Oct 30 23:02:44.242825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.242721    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"]
Oct 30 23:02:44.253628 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.253595    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0a807-openshift-default-66f955d779-qv6wr"]
Oct 30 23:02:44.375063 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.375034    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"]
Oct 30 23:02:44.375725 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.375708    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy"
Oct 30 23:02:44.375839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.375726    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy"
Oct 30 23:02:44.375839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.375764    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy"
Oct 30 23:02:44.375839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.375773    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy"
Oct 30 23:02:44.376010 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.375899    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy"
Oct 30 23:02:44.376010 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.375911    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" containerName="istio-proxy"
Oct 30 23:02:44.379954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.379927    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.400921 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.400892    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"]
Oct 30 23:02:44.481598 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.481512    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvc2n\" (UniqueName: \"kubernetes.io/projected/ae8655d1-7428-4ca1-b6ce-351692ba1815-kube-api-access-jvc2n\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.481771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.481746    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.481818 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.481788    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.583377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.583342    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.583575 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.583383    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.583653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.583596    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvc2n\" (UniqueName: \"kubernetes.io/projected/ae8655d1-7428-4ca1-b6ce-351692ba1815-kube-api-access-jvc2n\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.584318 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.584283    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.586435 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.586408    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.602873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.602834    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvc2n\" (UniqueName: \"kubernetes.io/projected/ae8655d1-7428-4ca1-b6ce-351692ba1815-kube-api-access-jvc2n\") pod \"maas-api-revoke-sub-3252ab44-pcgx9\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:44.702623 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:44.702587    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:45.020516 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:45.020463    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"]
Oct 30 23:02:45.021943 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:02:45.021915    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8655d1_7428_4ca1_b6ce_351692ba1815.slice/crio-ad9773e6cd36bc97db7b7a1939756948e60325732ff8f7dc64ab917e4e82a663 WatchSource:0}: Error finding container ad9773e6cd36bc97db7b7a1939756948e60325732ff8f7dc64ab917e4e82a663: Status 404 returned error can't find the container with id ad9773e6cd36bc97db7b7a1939756948e60325732ff8f7dc64ab917e4e82a663
Oct 30 23:02:45.204742 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:45.204704    4581 generic.go:356] "Generic (PLEG): container finished" podID="d4563a2d-4edd-40e7-a786-3fdcde95e0f5" containerID="154db057e611dd26637248bbd1e4e3f3f1afd2e27ab61ee283937fe79e414257" exitCode=0
Oct 30 23:02:45.205188 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:45.204750    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2" event={"ID":"d4563a2d-4edd-40e7-a786-3fdcde95e0f5","Type":"ContainerDied","Data":"154db057e611dd26637248bbd1e4e3f3f1afd2e27ab61ee283937fe79e414257"}
Oct 30 23:02:45.206448 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:45.206428    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9" event={"ID":"ae8655d1-7428-4ca1-b6ce-351692ba1815","Type":"ContainerStarted","Data":"ad9773e6cd36bc97db7b7a1939756948e60325732ff8f7dc64ab917e4e82a663"}
Oct 30 23:02:46.099203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.099167    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd29e823-643c-4ef9-b94f-dbffd2461749" path="/var/lib/kubelet/pods/dd29e823-643c-4ef9-b94f-dbffd2461749/volumes"
Oct 30 23:02:46.218541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.218505    4581 generic.go:356] "Generic (PLEG): container finished" podID="ae8655d1-7428-4ca1-b6ce-351692ba1815" containerID="e879d53ad563a70f12bfef12a9ba37e861adb326a9dbbe1cf3ba1253f0800950" exitCode=0
Oct 30 23:02:46.219034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.218599    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9" event={"ID":"ae8655d1-7428-4ca1-b6ce-351692ba1815","Type":"ContainerDied","Data":"e879d53ad563a70f12bfef12a9ba37e861adb326a9dbbe1cf3ba1253f0800950"}
Oct 30 23:02:46.397811 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.397777    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:46.515486 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.515451    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-xks-ca\") pod \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") "
Oct 30 23:02:46.515723 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.515516    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-kube-api-access-tfp8b\" (UniqueName: \"kubernetes.io/projected/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-kube-api-access-tfp8b\") pod \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") "
Oct 30 23:02:46.515723 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.515644    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-openshift-ca\") pod \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\" (UID: \"d4563a2d-4edd-40e7-a786-3fdcde95e0f5\") "
Oct 30 23:02:46.515979 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.515953    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-openshift-ca" pod "d4563a2d-4edd-40e7-a786-3fdcde95e0f5" (UID: "d4563a2d-4edd-40e7-a786-3fdcde95e0f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:46.516326 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.516307    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:46.517878 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.517850    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-xks-ca" pod "d4563a2d-4edd-40e7-a786-3fdcde95e0f5" (UID: "d4563a2d-4edd-40e7-a786-3fdcde95e0f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:46.517992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.517899    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-kube-api-access-tfp8b" pod "d4563a2d-4edd-40e7-a786-3fdcde95e0f5" (UID: "d4563a2d-4edd-40e7-a786-3fdcde95e0f5"). InnerVolumeSpecName "kube-api-access-tfp8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:46.617791 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.617698    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:46.617791 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:46.617729    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfp8b\" (UniqueName: \"kubernetes.io/projected/d4563a2d-4edd-40e7-a786-3fdcde95e0f5-kube-api-access-tfp8b\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.014181 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.014093    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:02:47.014181 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.014147    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:02:47.230059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.230030    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"
Oct 30 23:02:47.230587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.230066    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2" event={"ID":"d4563a2d-4edd-40e7-a786-3fdcde95e0f5","Type":"ContainerDied","Data":"4d53124bee2592dd278149224eb2b6c91e58f67ec5012fca1bef486694e2bc06"}
Oct 30 23:02:47.230587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.230125    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d53124bee2592dd278149224eb2b6c91e58f67ec5012fca1bef486694e2bc06"
Oct 30 23:02:47.395945 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.395912    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:47.428894 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.428862    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-openshift-ca\") pod \"ae8655d1-7428-4ca1-b6ce-351692ba1815\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") "
Oct 30 23:02:47.429077 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.428985    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8655d1-7428-4ca1-b6ce-351692ba1815-kube-api-access-jvc2n\" (UniqueName: \"kubernetes.io/projected/ae8655d1-7428-4ca1-b6ce-351692ba1815-kube-api-access-jvc2n\") pod \"ae8655d1-7428-4ca1-b6ce-351692ba1815\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") "
Oct 30 23:02:47.429124 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.429094    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-xks-ca\") pod \"ae8655d1-7428-4ca1-b6ce-351692ba1815\" (UID: \"ae8655d1-7428-4ca1-b6ce-351692ba1815\") "
Oct 30 23:02:47.429305 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.429276    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-openshift-ca" pod "ae8655d1-7428-4ca1-b6ce-351692ba1815" (UID: "ae8655d1-7428-4ca1-b6ce-351692ba1815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:47.429592 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.429572    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.431384 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.431352    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8655d1-7428-4ca1-b6ce-351692ba1815-kube-api-access-jvc2n" pod "ae8655d1-7428-4ca1-b6ce-351692ba1815" (UID: "ae8655d1-7428-4ca1-b6ce-351692ba1815"). InnerVolumeSpecName "kube-api-access-jvc2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:47.431384 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.431368    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-xks-ca" pod "ae8655d1-7428-4ca1-b6ce-351692ba1815" (UID: "ae8655d1-7428-4ca1-b6ce-351692ba1815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:47.531312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.531276    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvc2n\" (UniqueName: \"kubernetes.io/projected/ae8655d1-7428-4ca1-b6ce-351692ba1815-kube-api-access-jvc2n\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.531312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.531310    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8655d1-7428-4ca1-b6ce-351692ba1815-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.888097 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.888069    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 23:02:47.937091 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937055    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-socket\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937120    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istiod-ca-cert\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937173    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-certs\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937256    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-data\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937296    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-podinfo\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937398    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-socket" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937431    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-envoy\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937466    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-token\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-token\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937518    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-certs" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937596    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-credential-socket\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937625    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-data" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937627    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-kube-api-access-lv8bn\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-kube-api-access-lv8bn\") pod \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\" (UID: \"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1\") "
Oct 30 23:02:47.937670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.937647    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istiod-ca-cert" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:47.938406 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.938045    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-credential-socket" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.938750 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.938723    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.938839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.938753    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.938839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.938771    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.938839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.938785    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.938839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.938798    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:47.940385 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.940353    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-podinfo" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:02:47.940498 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.940387    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-kube-api-access-lv8bn" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "kube-api-access-lv8bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:47.940568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.940511    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-envoy" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:47.940568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:47.940520    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-token" pod "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" (UID: "ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:48.040626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.040591    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.040626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.040623    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.040626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.040634    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.040888 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.040646    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8bn\" (UniqueName: \"kubernetes.io/projected/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1-kube-api-access-lv8bn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:48.240292 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.240260    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"
Oct 30 23:02:48.240292 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.240272    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9" event={"ID":"ae8655d1-7428-4ca1-b6ce-351692ba1815","Type":"ContainerDied","Data":"ad9773e6cd36bc97db7b7a1939756948e60325732ff8f7dc64ab917e4e82a663"}
Oct 30 23:02:48.240808 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.240366    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9773e6cd36bc97db7b7a1939756948e60325732ff8f7dc64ab917e4e82a663"
Oct 30 23:02:48.242397 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.242364    4581 generic.go:356] "Generic (PLEG): container finished" podID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerID="6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028" exitCode=0
Oct 30 23:02:48.242549 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.242446    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"
Oct 30 23:02:48.242549 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.242477    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" event={"ID":"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1","Type":"ContainerDied","Data":"6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028"}
Oct 30 23:02:48.242549 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.242524    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr" event={"ID":"ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1","Type":"ContainerDied","Data":"68b4ddff9d7e3b332c394f68a633d26198171cefb52c6ec579d060abf119d51e"}
Oct 30 23:02:48.242699 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.242557    4581 scope.go:122] "RemoveContainer" containerID="6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028"
Oct 30 23:02:48.273963 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.273873    4581 scope.go:122] "RemoveContainer" containerID="cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5"
Oct 30 23:02:48.288802 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.288773    4581 scope.go:122] "RemoveContainer" containerID="6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028"
Oct 30 23:02:48.289182 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:48.289162    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028\": container with ID starting with 6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028 not found: ID does not exist" containerID="6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028"
Oct 30 23:02:48.289357 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.289192    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028"} err="failed to get container status \"6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028\": rpc error: code = NotFound desc = could not find container \"6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028\": container with ID starting with 6e6717aacdcbcc52cb727af272a41fac6976d6ce0051dcbb3d45198d37455028 not found: ID does not exist"
Oct 30 23:02:48.289357 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.289209    4581 scope.go:122] "RemoveContainer" containerID="cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5"
Oct 30 23:02:48.289545 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:48.289523    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5\": container with ID starting with cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5 not found: ID does not exist" containerID="cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5"
Oct 30 23:02:48.289612 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.289553    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5"} err="failed to get container status \"cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5\": rpc error: code = NotFound desc = could not find container \"cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5\": container with ID starting with cae76e41f3d5117849c00c53bbabb43ad2155ec7c84e46eecff82cbd4d8fcad5 not found: ID does not exist"
Oct 30 23:02:48.306190 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.306161    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"]
Oct 30 23:02:48.316393 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:48.316368    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6f2eba-openshift-default-5658f7689c-krxtr"]
Oct 30 23:02:49.314139 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314100    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"]
Oct 30 23:02:49.314771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314755    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy"
Oct 30 23:02:49.314826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314772    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy"
Oct 30 23:02:49.314826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314788    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy"
Oct 30 23:02:49.314826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314793    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy"
Oct 30 23:02:49.314826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314812    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ae8655d1-7428-4ca1-b6ce-351692ba1815" containerName="revoke-keys"
Oct 30 23:02:49.314826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314817    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8655d1-7428-4ca1-b6ce-351692ba1815" containerName="revoke-keys"
Oct 30 23:02:49.314826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314825    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d4563a2d-4edd-40e7-a786-3fdcde95e0f5" containerName="revoke-keys"
Oct 30 23:02:49.315087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314830    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4563a2d-4edd-40e7-a786-3fdcde95e0f5" containerName="revoke-keys"
Oct 30 23:02:49.315087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314925    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8655d1-7428-4ca1-b6ce-351692ba1815" containerName="revoke-keys"
Oct 30 23:02:49.315087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314933    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4563a2d-4edd-40e7-a786-3fdcde95e0f5" containerName="revoke-keys"
Oct 30 23:02:49.315087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314945    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy"
Oct 30 23:02:49.315087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.314953    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" containerName="istio-proxy"
Oct 30 23:02:49.320007 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.319985    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.333153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.333128    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:02:49.339510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.339481    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"]
Oct 30 23:02:49.343327 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.343298    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:02:49.343474 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.343362    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:02:49.343474 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.343422    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:02:49.360666 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.360629    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpzz\" (UniqueName: \"kubernetes.io/projected/9deebf80-7aab-41e2-a15f-19db5a5474ec-kube-api-access-ccpzz\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.360910 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.360886    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.361017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.360996    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.462274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.462199    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpzz\" (UniqueName: \"kubernetes.io/projected/9deebf80-7aab-41e2-a15f-19db5a5474ec-kube-api-access-ccpzz\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.462474 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.462425    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.462582 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.462481    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.463422 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.463387    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.468828 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.468805    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.479644 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.479618    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpzz\" (UniqueName: \"kubernetes.io/projected/9deebf80-7aab-41e2-a15f-19db5a5474ec-kube-api-access-ccpzz\") pod \"maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.635099 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.635066    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:49.960922 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:02:49.960886    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9deebf80_7aab_41e2_a15f_19db5a5474ec.slice/crio-75529a10a3f7c2a12b747749c4405709fbb307522e6e6dc0775a351d6cd6c3c2 WatchSource:0}: Error finding container 75529a10a3f7c2a12b747749c4405709fbb307522e6e6dc0775a351d6cd6c3c2: Status 404 returned error can't find the container with id 75529a10a3f7c2a12b747749c4405709fbb307522e6e6dc0775a351d6cd6c3c2
Oct 30 23:02:49.961761 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:49.961737    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"]
Oct 30 23:02:50.099494 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:50.099457    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1" path="/var/lib/kubelet/pods/ead9f4b6-1d08-4006-a8a9-e99ebcca9ac1/volumes"
Oct 30 23:02:50.269444 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:50.266073    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf" event={"ID":"9deebf80-7aab-41e2-a15f-19db5a5474ec","Type":"ContainerStarted","Data":"75529a10a3f7c2a12b747749c4405709fbb307522e6e6dc0775a351d6cd6c3c2"}
Oct 30 23:02:50.482192 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:50.482149    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"]
Oct 30 23:02:50.482673 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:50.482428    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy" containerID="cri-o://207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3" gracePeriod=30
Oct 30 23:02:51.276159 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:51.276053    4581 generic.go:356] "Generic (PLEG): container finished" podID="9deebf80-7aab-41e2-a15f-19db5a5474ec" containerID="bd98eb59fbcec91dcc8dff50b5df5fef9af74c736944efade192a76009ae772b" exitCode=0
Oct 30 23:02:51.276159 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:51.276146    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf" event={"ID":"9deebf80-7aab-41e2-a15f-19db5a5474ec","Type":"ContainerDied","Data":"bd98eb59fbcec91dcc8dff50b5df5fef9af74c736944efade192a76009ae772b"}
Oct 30 23:02:52.453038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.453009    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:52.508760 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.508720    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-xks-ca\") pod \"9deebf80-7aab-41e2-a15f-19db5a5474ec\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") "
Oct 30 23:02:52.508962 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.508881    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9deebf80-7aab-41e2-a15f-19db5a5474ec-kube-api-access-ccpzz\" (UniqueName: \"kubernetes.io/projected/9deebf80-7aab-41e2-a15f-19db5a5474ec-kube-api-access-ccpzz\") pod \"9deebf80-7aab-41e2-a15f-19db5a5474ec\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") "
Oct 30 23:02:52.509084 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.509061    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-openshift-ca\") pod \"9deebf80-7aab-41e2-a15f-19db5a5474ec\" (UID: \"9deebf80-7aab-41e2-a15f-19db5a5474ec\") "
Oct 30 23:02:52.509494 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.509458    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-openshift-ca" pod "9deebf80-7aab-41e2-a15f-19db5a5474ec" (UID: "9deebf80-7aab-41e2-a15f-19db5a5474ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:52.509968 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.509940    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:52.511301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.511272    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-xks-ca" pod "9deebf80-7aab-41e2-a15f-19db5a5474ec" (UID: "9deebf80-7aab-41e2-a15f-19db5a5474ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:52.511930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.511902    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deebf80-7aab-41e2-a15f-19db5a5474ec-kube-api-access-ccpzz" pod "9deebf80-7aab-41e2-a15f-19db5a5474ec" (UID: "9deebf80-7aab-41e2-a15f-19db5a5474ec"). InnerVolumeSpecName "kube-api-access-ccpzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:52.611311 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.611277    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9deebf80-7aab-41e2-a15f-19db5a5474ec-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:52.611311 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:52.611309    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccpzz\" (UniqueName: \"kubernetes.io/projected/9deebf80-7aab-41e2-a15f-19db5a5474ec-kube-api-access-ccpzz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:53.299052 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:53.299018    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"
Oct 30 23:02:53.299052 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:53.299041    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf" event={"ID":"9deebf80-7aab-41e2-a15f-19db5a5474ec","Type":"ContainerDied","Data":"75529a10a3f7c2a12b747749c4405709fbb307522e6e6dc0775a351d6cd6c3c2"}
Oct 30 23:02:53.299390 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:53.299094    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75529a10a3f7c2a12b747749c4405709fbb307522e6e6dc0775a351d6cd6c3c2"
Oct 30 23:02:54.925496 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.925463    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"]
Oct 30 23:02:54.926105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.926084    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9deebf80-7aab-41e2-a15f-19db5a5474ec" containerName="revoke-keys"
Oct 30 23:02:54.926105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.926099    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9deebf80-7aab-41e2-a15f-19db5a5474ec" containerName="revoke-keys"
Oct 30 23:02:54.926347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.926332    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9deebf80-7aab-41e2-a15f-19db5a5474ec" containerName="revoke-keys"
Oct 30 23:02:54.930200 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.930175    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:54.955088 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.955055    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:02:54.955445 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.955428    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:02:54.955630 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.955617    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:02:54.955831 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.955817    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:02:54.955984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:54.955842    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"]
Oct 30 23:02:55.047537 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.047485    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.047742 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.047649    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.047788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.047769    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7g4z\" (UniqueName: \"kubernetes.io/projected/ebdc3870-d2c2-400b-a663-8caaf51e37e2-kube-api-access-x7g4z\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.149555 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.149496    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.149729 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.149634    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.149729 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.149723    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7g4z\" (UniqueName: \"kubernetes.io/projected/ebdc3870-d2c2-400b-a663-8caaf51e37e2-kube-api-access-x7g4z\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.150460 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.150428    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.152789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.152752    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.167962 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.167928    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7g4z\" (UniqueName: \"kubernetes.io/projected/ebdc3870-d2c2-400b-a663-8caaf51e37e2-kube-api-access-x7g4z\") pod \"maas-api-revoke-sub-8b2c776d-9p6sf\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.245570 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.245467    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:55.778754 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.778720    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"]
Oct 30 23:02:55.803548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.803516    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 23:02:55.881725 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.881686    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277811fe-0837-4fcf-8c7e-9e174c3bf630-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277811fe-0837-4fcf-8c7e-9e174c3bf630-istiod-ca-cert\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.881928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.881789    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-certs\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.881928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.881847    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-envoy\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.881928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.881877    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-kube-api-access-b9lff\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-kube-api-access-b9lff\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.882097 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.881928    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-podinfo\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.882097 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.881990    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-credential-socket\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.882097 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882020    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-token\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-token\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.882097 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882051    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277811fe-0837-4fcf-8c7e-9e174c3bf630-istiod-ca-cert" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:55.882097 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882108    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-data\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.882407 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882153    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-certs" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:55.882407 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882320    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-socket\") pod \"277811fe-0837-4fcf-8c7e-9e174c3bf630\" (UID: \"277811fe-0837-4fcf-8c7e-9e174c3bf630\") "
Oct 30 23:02:55.882407 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882372    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-data" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:55.882590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882571    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-credential-socket" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:55.882922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882894    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277811fe-0837-4fcf-8c7e-9e174c3bf630-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:55.883038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882926    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:55.883038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882941    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:55.883038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.882954    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:55.883380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.883044    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-socket" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:55.885032 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.885001    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-podinfo" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:02:55.885141 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.885032    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-token" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:55.893301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.887872    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-envoy" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:55.893301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.888441    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-kube-api-access-b9lff" pod "277811fe-0837-4fcf-8c7e-9e174c3bf630" (UID: "277811fe-0837-4fcf-8c7e-9e174c3bf630"). InnerVolumeSpecName "kube-api-access-b9lff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:55.985336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.985286    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:55.985336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.985329    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9lff\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-kube-api-access-b9lff\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:55.985899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.985348    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:55.985899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.985362    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277811fe-0837-4fcf-8c7e-9e174c3bf630-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:55.985899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:55.985376    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277811fe-0837-4fcf-8c7e-9e174c3bf630-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:56.435306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.434642    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf" event={"ID":"ebdc3870-d2c2-400b-a663-8caaf51e37e2","Type":"ContainerStarted","Data":"9e025cfbd32d81d5375e90829b8126d9adaeb9e5285070abe64c4a04b1ada34e"}
Oct 30 23:02:56.435306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.434708    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf" event={"ID":"ebdc3870-d2c2-400b-a663-8caaf51e37e2","Type":"ContainerStarted","Data":"d4c0af25b5d23598ef90dde645a12ccc10d85b9f09b13fc01c0a08b2242cb311"}
Oct 30 23:02:56.445274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.439437    4581 generic.go:356] "Generic (PLEG): container finished" podID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerID="207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3" exitCode=0
Oct 30 23:02:56.445274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.439633    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" event={"ID":"277811fe-0837-4fcf-8c7e-9e174c3bf630","Type":"ContainerDied","Data":"207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3"}
Oct 30 23:02:56.445274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.439671    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7" event={"ID":"277811fe-0837-4fcf-8c7e-9e174c3bf630","Type":"ContainerDied","Data":"01610faf1612793d0b9add4a066195e300a5028f308619c9b44ae180196286b0"}
Oct 30 23:02:56.445274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.439693    4581 scope.go:122] "RemoveContainer" containerID="207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3"
Oct 30 23:02:56.445274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.439833    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"
Oct 30 23:02:56.459977 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.459949    4581 scope.go:122] "RemoveContainer" containerID="207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3"
Oct 30 23:02:56.460339 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:02:56.460315    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3\": container with ID starting with 207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3 not found: ID does not exist" containerID="207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3"
Oct 30 23:02:56.460436 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.460348    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3"} err="failed to get container status \"207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3\": rpc error: code = NotFound desc = could not find container \"207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3\": container with ID starting with 207abf03c90617a30048612db3147f03848db4f1c6fe48d090f94811986a08d3 not found: ID does not exist"
Oct 30 23:02:56.478004 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.477935    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf" podStartSLOduration=2.477918212 podStartE2EDuration="2.477918212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:56.476114794 +0000 UTC m=+3360.515053680" watchObservedRunningTime="2026-10-30 23:02:56.477918212 +0000 UTC m=+3360.516857099"
Oct 30 23:02:56.580254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.574792    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"]
Oct 30 23:02:56.600274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:56.597198    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6af86e-openshift-default-5bd446b89c-2xvj7"]
Oct 30 23:02:57.477241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:57.470712    4581 generic.go:356] "Generic (PLEG): container finished" podID="ebdc3870-d2c2-400b-a663-8caaf51e37e2" containerID="9e025cfbd32d81d5375e90829b8126d9adaeb9e5285070abe64c4a04b1ada34e" exitCode=0
Oct 30 23:02:57.477241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:57.471173    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf" event={"ID":"ebdc3870-d2c2-400b-a663-8caaf51e37e2","Type":"ContainerDied","Data":"9e025cfbd32d81d5375e90829b8126d9adaeb9e5285070abe64c4a04b1ada34e"}
Oct 30 23:02:58.099504 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.099469    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" path="/var/lib/kubelet/pods/277811fe-0837-4fcf-8c7e-9e174c3bf630/volumes"
Oct 30 23:02:58.713644 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.713616    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:58.748505 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.748409    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-openshift-ca\") pod \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") "
Oct 30 23:02:58.748677 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.748579    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdc3870-d2c2-400b-a663-8caaf51e37e2-kube-api-access-x7g4z\" (UniqueName: \"kubernetes.io/projected/ebdc3870-d2c2-400b-a663-8caaf51e37e2-kube-api-access-x7g4z\") pod \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") "
Oct 30 23:02:58.748739 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.748694    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-xks-ca\") pod \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\" (UID: \"ebdc3870-d2c2-400b-a663-8caaf51e37e2\") "
Oct 30 23:02:58.764649 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.748802    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-openshift-ca" pod "ebdc3870-d2c2-400b-a663-8caaf51e37e2" (UID: "ebdc3870-d2c2-400b-a663-8caaf51e37e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:58.799267 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.798509    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdc3870-d2c2-400b-a663-8caaf51e37e2-kube-api-access-x7g4z" pod "ebdc3870-d2c2-400b-a663-8caaf51e37e2" (UID: "ebdc3870-d2c2-400b-a663-8caaf51e37e2"). InnerVolumeSpecName "kube-api-access-x7g4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:58.799267 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.798853    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7g4z\" (UniqueName: \"kubernetes.io/projected/ebdc3870-d2c2-400b-a663-8caaf51e37e2-kube-api-access-x7g4z\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:58.799267 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.798875    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:58.823283 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.822742    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-xks-ca" pod "ebdc3870-d2c2-400b-a663-8caaf51e37e2" (UID: "ebdc3870-d2c2-400b-a663-8caaf51e37e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:58.899826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:58.899784    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc3870-d2c2-400b-a663-8caaf51e37e2-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:02:59.496053 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:59.496016    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf" event={"ID":"ebdc3870-d2c2-400b-a663-8caaf51e37e2","Type":"ContainerDied","Data":"d4c0af25b5d23598ef90dde645a12ccc10d85b9f09b13fc01c0a08b2242cb311"}
Oct 30 23:02:59.496053 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:59.496046    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"
Oct 30 23:02:59.496349 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:02:59.496070    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4c0af25b5d23598ef90dde645a12ccc10d85b9f09b13fc01c0a08b2242cb311"
Oct 30 23:03:02.504614 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:02.504573    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"]
Oct 30 23:03:02.505492 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:02.505455    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d" podUID="0cae1609-18f2-422c-81b5-cc7e850ccc0c" containerName="payload-pre-processing" containerID="cri-o://2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493" gracePeriod=30
Oct 30 23:03:02.600899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:02.600867    4581 scope.go:122] "RemoveContainer" containerID="fe18f5f560eebd7feda298ddd71cf603babc554c76c42dac0a2181db586c15c7"
Oct 30 23:03:02.947290 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:02.947209    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:03:02.971931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:02.971902    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cae1609-18f2-422c-81b5-cc7e850ccc0c-kube-api-access-sz9s9\" (UniqueName: \"kubernetes.io/projected/0cae1609-18f2-422c-81b5-cc7e850ccc0c-kube-api-access-sz9s9\") pod \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\" (UID: \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\") "
Oct 30 23:03:02.972139 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:02.972091    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cae1609-18f2-422c-81b5-cc7e850ccc0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cae1609-18f2-422c-81b5-cc7e850ccc0c-plugins-config-volume\") pod \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\" (UID: \"0cae1609-18f2-422c-81b5-cc7e850ccc0c\") "
Oct 30 23:03:02.972563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:02.972531    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cae1609-18f2-422c-81b5-cc7e850ccc0c-plugins-config-volume" pod "0cae1609-18f2-422c-81b5-cc7e850ccc0c" (UID: "0cae1609-18f2-422c-81b5-cc7e850ccc0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:02.974821 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:02.974795    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cae1609-18f2-422c-81b5-cc7e850ccc0c-kube-api-access-sz9s9" pod "0cae1609-18f2-422c-81b5-cc7e850ccc0c" (UID: "0cae1609-18f2-422c-81b5-cc7e850ccc0c"). InnerVolumeSpecName "kube-api-access-sz9s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:03.074203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.074159    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cae1609-18f2-422c-81b5-cc7e850ccc0c-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:03.074203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.074189    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9s9\" (UniqueName: \"kubernetes.io/projected/0cae1609-18f2-422c-81b5-cc7e850ccc0c-kube-api-access-sz9s9\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:03.126002 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.125961    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"]
Oct 30 23:03:03.130448 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.130421    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cdfa86-kmxjf"]
Oct 30 23:03:03.551696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.551656    4581 generic.go:356] "Generic (PLEG): container finished" podID="0cae1609-18f2-422c-81b5-cc7e850ccc0c" containerID="2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493" exitCode=0
Oct 30 23:03:03.551696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.551736    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"
Oct 30 23:03:03.551696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.551763    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d" event={"ID":"0cae1609-18f2-422c-81b5-cc7e850ccc0c","Type":"ContainerDied","Data":"2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493"}
Oct 30 23:03:03.551696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.551878    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d" event={"ID":"0cae1609-18f2-422c-81b5-cc7e850ccc0c","Type":"ContainerDied","Data":"42c00db7aeadef06f5733b46f5e02de390cd4981b87e8ecdd0bd61a44d361296"}
Oct 30 23:03:03.551696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.551926    4581 scope.go:122] "RemoveContainer" containerID="2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493"
Oct 30 23:03:03.569571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.569549    4581 scope.go:122] "RemoveContainer" containerID="2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493"
Oct 30 23:03:03.569960 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:03.569934    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493\": container with ID starting with 2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493 not found: ID does not exist" containerID="2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493"
Oct 30 23:03:03.570057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.569973    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493"} err="failed to get container status \"2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493\": rpc error: code = NotFound desc = could not find container \"2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493\": container with ID starting with 2d457a25a69dbdbc6f8b13bb15f744b98ef4a78a4dbb398669578e456d1f8493 not found: ID does not exist"
Oct 30 23:03:03.603309 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.603272    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"]
Oct 30 23:03:03.611311 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:03.611282    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cdfa86-5799cbf495-h927d"]
Oct 30 23:03:04.100908 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:04.100873    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cae1609-18f2-422c-81b5-cc7e850ccc0c" path="/var/lib/kubelet/pods/0cae1609-18f2-422c-81b5-cc7e850ccc0c/volumes"
Oct 30 23:03:04.101424 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:04.101406    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9deebf80-7aab-41e2-a15f-19db5a5474ec" path="/var/lib/kubelet/pods/9deebf80-7aab-41e2-a15f-19db5a5474ec/volumes"
Oct 30 23:03:06.595924 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:06.595879    4581 generic.go:356] "Generic (PLEG): container finished" podID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerID="196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9" exitCode=0
Oct 30 23:03:06.596519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:06.595972    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" event={"ID":"b3da96f9-b10e-4413-995f-0a3dc89511d4","Type":"ContainerDied","Data":"196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9"}
Oct 30 23:03:06.596593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:06.596533    4581 scope.go:122] "RemoveContainer" containerID="196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9"
Oct 30 23:03:06.599025 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:06.598986    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:03:06.599177 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:06.599074    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:03:06.599177 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:06.599110    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:03:07.310487 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:07.310417    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"]
Oct 30 23:03:07.328590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:07.328549    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d9f1c2-54g9v"]
Oct 30 23:03:07.610519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:07.610487    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" event={"ID":"b3da96f9-b10e-4413-995f-0a3dc89511d4","Type":"ContainerStarted","Data":"f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4"}
Oct 30 23:03:07.744300 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:07.744264    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:03:07.745273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:07.745244    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 23:03:07.745452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:07.745309    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 23:03:08.100503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:08.100422    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7fd0369-9802-468e-80d2-0dde78473b51" path="/var/lib/kubelet/pods/b7fd0369-9802-468e-80d2-0dde78473b51/volumes"
Oct 30 23:03:08.745096 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:08.745060    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 23:03:08.745621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:08.745113    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 23:03:09.744463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:09.744427    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 23:03:09.744694 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:09.744484    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 23:03:10.761245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:10.745314    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 23:03:10.761245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:10.745374    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 23:03:10.821736 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:10.821695    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:03:10.833829 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:10.833803    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:03:10.948547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:10.948508    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"]
Oct 30 23:03:10.948761 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:10.948707    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy" containerID="cri-o://e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca" gracePeriod=30
Oct 30 23:03:11.512005 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.511969    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"]
Oct 30 23:03:11.512621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512607    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0cae1609-18f2-422c-81b5-cc7e850ccc0c" containerName="payload-pre-processing"
Oct 30 23:03:11.512679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512622    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cae1609-18f2-422c-81b5-cc7e850ccc0c" containerName="payload-pre-processing"
Oct 30 23:03:11.512679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512638    4581 cpu_manager.go:414] "containerMap: removing container" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy"
Oct 30 23:03:11.512679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512643    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy"
Oct 30 23:03:11.512679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512661    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ebdc3870-d2c2-400b-a663-8caaf51e37e2" containerName="revoke-keys"
Oct 30 23:03:11.512679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512665    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdc3870-d2c2-400b-a663-8caaf51e37e2" containerName="revoke-keys"
Oct 30 23:03:11.512848 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512758    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cae1609-18f2-422c-81b5-cc7e850ccc0c" containerName="payload-pre-processing"
Oct 30 23:03:11.512848 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512769    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="277811fe-0837-4fcf-8c7e-9e174c3bf630" containerName="istio-proxy"
Oct 30 23:03:11.512848 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.512777    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdc3870-d2c2-400b-a663-8caaf51e37e2" containerName="revoke-keys"
Oct 30 23:03:11.516753 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.516733    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.536589 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.536566    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"]
Oct 30 23:03:11.541404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.541381    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:11.541404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.541398    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:11.617601 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.617565    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:11.617798 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.617719    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:03:11.698540 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.698502    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.698774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.698752    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.698849 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.698793    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6phv\" (UniqueName: \"kubernetes.io/projected/986c2252-d28e-4d1f-a6ae-42954d85e70b-kube-api-access-p6phv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.744382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.744345    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 23:03:11.744587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.744409    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 23:03:11.800671 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.800572    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.801165 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.800745    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.801165 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.800767    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6phv\" (UniqueName: \"kubernetes.io/projected/986c2252-d28e-4d1f-a6ae-42954d85e70b-kube-api-access-p6phv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.801458 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.801435    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.803158 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.803134    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.818300 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.818276    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6phv\" (UniqueName: \"kubernetes.io/projected/986c2252-d28e-4d1f-a6ae-42954d85e70b-kube-api-access-p6phv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:11.831313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:11.831288    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:12.186539 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:12.186506    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986c2252_d28e_4d1f_a6ae_42954d85e70b.slice/crio-ae233a7737c235d55b60781672b45aeba55cf1ce7cf4d4b2df46e631c4a8c4c4 WatchSource:0}: Error finding container ae233a7737c235d55b60781672b45aeba55cf1ce7cf4d4b2df46e631c4a8c4c4: Status 404 returned error can't find the container with id ae233a7737c235d55b60781672b45aeba55cf1ce7cf4d4b2df46e631c4a8c4c4
Oct 30 23:03:12.190006 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:12.189977    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"]
Oct 30 23:03:12.689824 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:12.689773    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk" event={"ID":"986c2252-d28e-4d1f-a6ae-42954d85e70b","Type":"ContainerStarted","Data":"df6c89da0061c5051b0aa7ac4c593d75fb3dbee5c6d26699485a2835f1f1fdc6"}
Oct 30 23:03:12.690010 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:12.689844    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk" event={"ID":"986c2252-d28e-4d1f-a6ae-42954d85e70b","Type":"ContainerStarted","Data":"ae233a7737c235d55b60781672b45aeba55cf1ce7cf4d4b2df46e631c4a8c4c4"}
Oct 30 23:03:12.727591 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:12.727541    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk" podStartSLOduration=1.7275261400000002 podStartE2EDuration="1.72752614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:12.724104112 +0000 UTC m=+3376.763042997" watchObservedRunningTime="2026-10-30 23:03:12.72752614 +0000 UTC m=+3376.766465025"
Oct 30 23:03:12.744653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:12.744618    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 23:03:12.744826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:12.744679    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 23:03:13.701015 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:13.700970    4581 generic.go:356] "Generic (PLEG): container finished" podID="986c2252-d28e-4d1f-a6ae-42954d85e70b" containerID="df6c89da0061c5051b0aa7ac4c593d75fb3dbee5c6d26699485a2835f1f1fdc6" exitCode=0
Oct 30 23:03:13.701538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:13.701361    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk" event={"ID":"986c2252-d28e-4d1f-a6ae-42954d85e70b","Type":"ContainerDied","Data":"df6c89da0061c5051b0aa7ac4c593d75fb3dbee5c6d26699485a2835f1f1fdc6"}
Oct 30 23:03:13.750401 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:13.750319    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:03:13.751948 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:13.751915    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:03:14.879762 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:14.879726    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:14.945089 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:14.945053    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986c2252-d28e-4d1f-a6ae-42954d85e70b-kube-api-access-p6phv\" (UniqueName: \"kubernetes.io/projected/986c2252-d28e-4d1f-a6ae-42954d85e70b-kube-api-access-p6phv\") pod \"986c2252-d28e-4d1f-a6ae-42954d85e70b\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") "
Oct 30 23:03:14.945303 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:14.945128    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-openshift-ca\") pod \"986c2252-d28e-4d1f-a6ae-42954d85e70b\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") "
Oct 30 23:03:14.945303 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:14.945248    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-xks-ca\") pod \"986c2252-d28e-4d1f-a6ae-42954d85e70b\" (UID: \"986c2252-d28e-4d1f-a6ae-42954d85e70b\") "
Oct 30 23:03:14.945921 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:14.945882    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-openshift-ca" pod "986c2252-d28e-4d1f-a6ae-42954d85e70b" (UID: "986c2252-d28e-4d1f-a6ae-42954d85e70b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:14.948374 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:14.948345    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986c2252-d28e-4d1f-a6ae-42954d85e70b-kube-api-access-p6phv" pod "986c2252-d28e-4d1f-a6ae-42954d85e70b" (UID: "986c2252-d28e-4d1f-a6ae-42954d85e70b"). InnerVolumeSpecName "kube-api-access-p6phv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:14.948374 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:14.948363    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-xks-ca" pod "986c2252-d28e-4d1f-a6ae-42954d85e70b" (UID: "986c2252-d28e-4d1f-a6ae-42954d85e70b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:15.046695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.046615    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6phv\" (UniqueName: \"kubernetes.io/projected/986c2252-d28e-4d1f-a6ae-42954d85e70b-kube-api-access-p6phv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:15.046695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.046642    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:15.046695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.046654    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986c2252-d28e-4d1f-a6ae-42954d85e70b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:15.241518 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.241485    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"]
Oct 30 23:03:15.241793 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.241766    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy" containerID="cri-o://b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a" gracePeriod=30
Oct 30 23:03:15.722685 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.722642    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"
Oct 30 23:03:15.722685 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.722675    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk" event={"ID":"986c2252-d28e-4d1f-a6ae-42954d85e70b","Type":"ContainerDied","Data":"ae233a7737c235d55b60781672b45aeba55cf1ce7cf4d4b2df46e631c4a8c4c4"}
Oct 30 23:03:15.722928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.722725    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae233a7737c235d55b60781672b45aeba55cf1ce7cf4d4b2df46e631c4a8c4c4"
Oct 30 23:03:15.948043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.948005    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"]
Oct 30 23:03:15.948970 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.948948    4581 cpu_manager.go:414] "containerMap: removing container" podUID="986c2252-d28e-4d1f-a6ae-42954d85e70b" containerName="revoke-keys"
Oct 30 23:03:15.949079 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.948972    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986c2252-d28e-4d1f-a6ae-42954d85e70b" containerName="revoke-keys"
Oct 30 23:03:15.949254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.949241    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="986c2252-d28e-4d1f-a6ae-42954d85e70b" containerName="revoke-keys"
Oct 30 23:03:15.953525 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.953506    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:15.967529 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.967503    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:15.967529 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.967517    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:15.967718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.967559    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:03:15.967718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.967587    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:15.969042 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.968756    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cst\" (UniqueName: \"kubernetes.io/projected/21135552-aa62-4db2-a43b-f9480fbbc6bd-kube-api-access-x7cst\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:15.969170 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.969040    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:15.969262 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.969231    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:15.973158 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:15.973104    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"]
Oct 30 23:03:16.070187 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.070155    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cst\" (UniqueName: \"kubernetes.io/projected/21135552-aa62-4db2-a43b-f9480fbbc6bd-kube-api-access-x7cst\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:16.070453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.070326    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:16.070453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.070444    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:16.071073 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.071048    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:16.072839 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.072816    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:16.088356 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.088328    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cst\" (UniqueName: \"kubernetes.io/projected/21135552-aa62-4db2-a43b-f9480fbbc6bd-kube-api-access-x7cst\") pod \"maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:16.268931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.268843    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:16.578861 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.578829    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"]
Oct 30 23:03:16.579354 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:16.579326    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21135552_aa62_4db2_a43b_f9480fbbc6bd.slice/crio-1bd6aa8fac880b66cfbeca655d099b53f402d2c7328831f83f63c6c01751333c WatchSource:0}: Error finding container 1bd6aa8fac880b66cfbeca655d099b53f402d2c7328831f83f63c6c01751333c: Status 404 returned error can't find the container with id 1bd6aa8fac880b66cfbeca655d099b53f402d2c7328831f83f63c6c01751333c
Oct 30 23:03:16.679277 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.679249    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 23:03:16.736000 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.735963    4581 generic.go:356] "Generic (PLEG): container finished" podID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerID="e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca" exitCode=0
Oct 30 23:03:16.736363 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.736332    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" event={"ID":"c069b4a9-3dd5-4144-a15e-e3df78c5f554","Type":"ContainerDied","Data":"e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca"}
Oct 30 23:03:16.736594 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.736565    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw" event={"ID":"c069b4a9-3dd5-4144-a15e-e3df78c5f554","Type":"ContainerDied","Data":"10b1181aa274a156f3de90a390846ad5a9dc598d8ded3e0b81b8917a7315f36e"}
Oct 30 23:03:16.736717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.736616    4581 scope.go:122] "RemoveContainer" containerID="e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca"
Oct 30 23:03:16.736788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.736746    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"
Oct 30 23:03:16.740146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.740117    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx" event={"ID":"21135552-aa62-4db2-a43b-f9480fbbc6bd","Type":"ContainerStarted","Data":"1bd6aa8fac880b66cfbeca655d099b53f402d2c7328831f83f63c6c01751333c"}
Oct 30 23:03:16.766255 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.766198    4581 scope.go:122] "RemoveContainer" containerID="a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c"
Oct 30 23:03:16.779466 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.779384    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istiod-ca-cert\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.779466 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.779461    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-envoy\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.779665 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.779532    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-kube-api-access-59h8s\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-kube-api-access-59h8s\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.779665 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.779628    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-certs\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.780074 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780047    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-data\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.780197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780146    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-certs" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:16.780197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780184    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istiod-ca-cert" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:16.780360 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780312    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-data" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:16.780412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780365    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-socket\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.780412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780395    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-podinfo\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.780510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780460    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-token\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-token\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.780611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780585    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-socket" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:16.780899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.780879    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-credential-socket\") pod \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\" (UID: \"c069b4a9-3dd5-4144-a15e-e3df78c5f554\") "
Oct 30 23:03:16.781742 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.781696    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-credential-socket" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:16.782468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.782448    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:16.782608 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.782594    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:16.782703 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.782691    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:16.782792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.782779    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:16.782884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.782872    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:16.783525 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.783488    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-podinfo" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:16.787517 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.784369    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-envoy" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:16.787783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.785164    4581 scope.go:122] "RemoveContainer" containerID="e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca"
Oct 30 23:03:16.788620 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:16.788554    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca\": container with ID starting with e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca not found: ID does not exist" containerID="e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca"
Oct 30 23:03:16.788858 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.788790    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca"} err="failed to get container status \"e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca\": rpc error: code = NotFound desc = could not find container \"e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca\": container with ID starting with e0f548295c4c72af1518c989b0a67ebd6f80b140c40cc09120e8459293d18eca not found: ID does not exist"
Oct 30 23:03:16.788858 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.788833    4581 scope.go:122] "RemoveContainer" containerID="a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c"
Oct 30 23:03:16.802262 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:16.802181    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c\": container with ID starting with a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c not found: ID does not exist" containerID="a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c"
Oct 30 23:03:16.802262 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.802247    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c"} err="failed to get container status \"a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c\": rpc error: code = NotFound desc = could not find container \"a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c\": container with ID starting with a8710fa2776d5a86a0c2de728f4c4734f34f5a9b0a8518ce55973f99a41b6b0c not found: ID does not exist"
Oct 30 23:03:16.803837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.803785    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-token" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:16.804144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.804115    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-kube-api-access-59h8s" pod "c069b4a9-3dd5-4144-a15e-e3df78c5f554" (UID: "c069b4a9-3dd5-4144-a15e-e3df78c5f554"). InnerVolumeSpecName "kube-api-access-59h8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:16.883985 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.883954    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:16.883985 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.883985    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59h8s\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-kube-api-access-59h8s\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:16.884179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.883994    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:16.884179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:16.884003    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c069b4a9-3dd5-4144-a15e-e3df78c5f554-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:17.080825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:17.080795    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"]
Oct 30 23:03:17.093184 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:17.093150    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdfa86-openshift-default-77bdfd8797-52tjw"]
Oct 30 23:03:17.756134 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:17.756092    4581 generic.go:356] "Generic (PLEG): container finished" podID="21135552-aa62-4db2-a43b-f9480fbbc6bd" containerID="f78f9d0b1a8750a1594cb29e9d8519de0c2c1f9bff2e7b6e36ebff33433b0729" exitCode=0
Oct 30 23:03:17.756656 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:17.756470    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx" event={"ID":"21135552-aa62-4db2-a43b-f9480fbbc6bd","Type":"ContainerDied","Data":"f78f9d0b1a8750a1594cb29e9d8519de0c2c1f9bff2e7b6e36ebff33433b0729"}
Oct 30 23:03:18.031010 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:18.030915    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"]
Oct 30 23:03:18.035186 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:18.035152    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb67f98-w58lr"]
Oct 30 23:03:18.097971 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:18.097937    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c37a12c-492f-404a-9582-c409842fb43d" path="/var/lib/kubelet/pods/7c37a12c-492f-404a-9582-c409842fb43d/volumes"
Oct 30 23:03:18.098500 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:18.098482    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" path="/var/lib/kubelet/pods/c069b4a9-3dd5-4144-a15e-e3df78c5f554/volumes"
Oct 30 23:03:18.878358 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:18.878306    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:03:18.878547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:18.878412    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:03:18.948083 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:18.948053    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:19.011694 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.011652    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21135552-aa62-4db2-a43b-f9480fbbc6bd-kube-api-access-x7cst\" (UniqueName: \"kubernetes.io/projected/21135552-aa62-4db2-a43b-f9480fbbc6bd-kube-api-access-x7cst\") pod \"21135552-aa62-4db2-a43b-f9480fbbc6bd\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") "
Oct 30 23:03:19.011895 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.011766    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-openshift-ca\") pod \"21135552-aa62-4db2-a43b-f9480fbbc6bd\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") "
Oct 30 23:03:19.011973 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.011891    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-xks-ca\") pod \"21135552-aa62-4db2-a43b-f9480fbbc6bd\" (UID: \"21135552-aa62-4db2-a43b-f9480fbbc6bd\") "
Oct 30 23:03:19.012145 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.012114    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-openshift-ca" pod "21135552-aa62-4db2-a43b-f9480fbbc6bd" (UID: "21135552-aa62-4db2-a43b-f9480fbbc6bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:19.012581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.012541    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.014052 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.014022    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-xks-ca" pod "21135552-aa62-4db2-a43b-f9480fbbc6bd" (UID: "21135552-aa62-4db2-a43b-f9480fbbc6bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:19.014138 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.014062    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21135552-aa62-4db2-a43b-f9480fbbc6bd-kube-api-access-x7cst" pod "21135552-aa62-4db2-a43b-f9480fbbc6bd" (UID: "21135552-aa62-4db2-a43b-f9480fbbc6bd"). InnerVolumeSpecName "kube-api-access-x7cst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:19.114180 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.114145    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21135552-aa62-4db2-a43b-f9480fbbc6bd-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.114180 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.114174    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7cst\" (UniqueName: \"kubernetes.io/projected/21135552-aa62-4db2-a43b-f9480fbbc6bd-kube-api-access-x7cst\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:19.776409 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.776369    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"
Oct 30 23:03:19.776598 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.776413    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx" event={"ID":"21135552-aa62-4db2-a43b-f9480fbbc6bd","Type":"ContainerDied","Data":"1bd6aa8fac880b66cfbeca655d099b53f402d2c7328831f83f63c6c01751333c"}
Oct 30 23:03:19.776598 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:19.776462    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bd6aa8fac880b66cfbeca655d099b53f402d2c7328831f83f63c6c01751333c"
Oct 30 23:03:20.016388 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.016346    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"]
Oct 30 23:03:20.016643 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.016600    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt" podUID="84821dee-7355-48ea-b90e-c2b8efb7ef34" containerName="payload-processing" containerID="cri-o://b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d" gracePeriod=30
Oct 30 23:03:20.035976 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.035895    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"]
Oct 30 23:03:20.036136 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.036110    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2" podUID="0aa77793-4b17-4cc7-8731-fcfcfc0cec75" containerName="payload-pre-processing" containerID="cri-o://2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b" gracePeriod=30
Oct 30 23:03:20.356480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.356450    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:03:20.480565 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.476389    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:03:20.539086 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.539053    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84821dee-7355-48ea-b90e-c2b8efb7ef34-kube-api-access-668ch\" (UniqueName: \"kubernetes.io/projected/84821dee-7355-48ea-b90e-c2b8efb7ef34-kube-api-access-668ch\") pod \"84821dee-7355-48ea-b90e-c2b8efb7ef34\" (UID: \"84821dee-7355-48ea-b90e-c2b8efb7ef34\") "
Oct 30 23:03:20.539314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.539142    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84821dee-7355-48ea-b90e-c2b8efb7ef34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84821dee-7355-48ea-b90e-c2b8efb7ef34-plugins-config-volume\") pod \"84821dee-7355-48ea-b90e-c2b8efb7ef34\" (UID: \"84821dee-7355-48ea-b90e-c2b8efb7ef34\") "
Oct 30 23:03:20.539314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.539278    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-kube-api-access-945v4\" (UniqueName: \"kubernetes.io/projected/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-kube-api-access-945v4\") pod \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\" (UID: \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\") "
Oct 30 23:03:20.539462 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.539342    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-plugins-config-volume\") pod \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\" (UID: \"0aa77793-4b17-4cc7-8731-fcfcfc0cec75\") "
Oct 30 23:03:20.539889 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.539776    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84821dee-7355-48ea-b90e-c2b8efb7ef34-plugins-config-volume" pod "84821dee-7355-48ea-b90e-c2b8efb7ef34" (UID: "84821dee-7355-48ea-b90e-c2b8efb7ef34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:20.540037 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.539965    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84821dee-7355-48ea-b90e-c2b8efb7ef34-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.540515 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.540439    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-plugins-config-volume" pod "0aa77793-4b17-4cc7-8731-fcfcfc0cec75" (UID: "0aa77793-4b17-4cc7-8731-fcfcfc0cec75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:20.542341 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.542316    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-kube-api-access-945v4" pod "0aa77793-4b17-4cc7-8731-fcfcfc0cec75" (UID: "0aa77793-4b17-4cc7-8731-fcfcfc0cec75"). InnerVolumeSpecName "kube-api-access-945v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:20.543796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.543774    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84821dee-7355-48ea-b90e-c2b8efb7ef34-kube-api-access-668ch" pod "84821dee-7355-48ea-b90e-c2b8efb7ef34" (UID: "84821dee-7355-48ea-b90e-c2b8efb7ef34"). InnerVolumeSpecName "kube-api-access-668ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:20.615768 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.615734    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"]
Oct 30 23:03:20.623141 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.623108    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76d54a-jl5nk"]
Oct 30 23:03:20.664254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.664184    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-945v4\" (UniqueName: \"kubernetes.io/projected/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-kube-api-access-945v4\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.664254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.664241    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa77793-4b17-4cc7-8731-fcfcfc0cec75-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.664254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.664259    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668ch\" (UniqueName: \"kubernetes.io/projected/84821dee-7355-48ea-b90e-c2b8efb7ef34-kube-api-access-668ch\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.681526 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.680776    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"]
Oct 30 23:03:20.697856 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.697821    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"]
Oct 30 23:03:20.727712 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.727682    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 23:03:20.765116 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765082    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-token\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-token\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.765314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765234    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-socket\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.765314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765267    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-certs\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.765413 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765329    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4b4443-3f69-43a0-9a55-118b93e68702-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4b4443-3f69-43a0-9a55-118b93e68702-istiod-ca-cert\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.765413 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765394    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-envoy\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.765541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765476    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-data\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.765541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765523    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-kube-api-access-v7kv2\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-kube-api-access-v7kv2\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.765647 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765580    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-podinfo\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.765647 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765628    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-credential-socket\") pod \"ae4b4443-3f69-43a0-9a55-118b93e68702\" (UID: \"ae4b4443-3f69-43a0-9a55-118b93e68702\") "
Oct 30 23:03:20.766121 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765621    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-certs" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:20.766121 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765714    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-data" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:20.766121 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.765546    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-socket" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:20.766121 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.766077    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-credential-socket" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:20.766121 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.766099    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4b4443-3f69-43a0-9a55-118b93e68702-istiod-ca-cert" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:20.766548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.766298    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.766548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.766317    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.766548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.766331    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.766548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.766344    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.766548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.766358    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4b4443-3f69-43a0-9a55-118b93e68702-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.767551 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.767525    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-token" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:20.768153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.768134    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-kube-api-access-v7kv2" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "kube-api-access-v7kv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:20.768317 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.768296    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-envoy" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:20.768377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.768307    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-podinfo" pod "ae4b4443-3f69-43a0-9a55-118b93e68702" (UID: "ae4b4443-3f69-43a0-9a55-118b93e68702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:20.786153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.786120    4581 generic.go:356] "Generic (PLEG): container finished" podID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerID="b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a" exitCode=0
Oct 30 23:03:20.786380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.786180    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"
Oct 30 23:03:20.786380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.786229    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" event={"ID":"ae4b4443-3f69-43a0-9a55-118b93e68702","Type":"ContainerDied","Data":"b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a"}
Oct 30 23:03:20.786380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.786296    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql" event={"ID":"ae4b4443-3f69-43a0-9a55-118b93e68702","Type":"ContainerDied","Data":"3f6d3670f06dac0b25f3c50808293f2aee72bfe40bc5522a0766c19179248989"}
Oct 30 23:03:20.786380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.786330    4581 scope.go:122] "RemoveContainer" containerID="b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a"
Oct 30 23:03:20.788131 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.788112    4581 generic.go:356] "Generic (PLEG): container finished" podID="0aa77793-4b17-4cc7-8731-fcfcfc0cec75" containerID="2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b" exitCode=0
Oct 30 23:03:20.788335 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.788303    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"
Oct 30 23:03:20.788424 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.788405    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2" event={"ID":"0aa77793-4b17-4cc7-8731-fcfcfc0cec75","Type":"ContainerDied","Data":"2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b"}
Oct 30 23:03:20.788485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.788440    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2" event={"ID":"0aa77793-4b17-4cc7-8731-fcfcfc0cec75","Type":"ContainerDied","Data":"7cc212111bd3cf295399a56c731a41743734fd2b4ef8b1ba5148006928c4f259"}
Oct 30 23:03:20.790269 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.790240    4581 generic.go:356] "Generic (PLEG): container finished" podID="84821dee-7355-48ea-b90e-c2b8efb7ef34" containerID="b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d" exitCode=0
Oct 30 23:03:20.790362 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.790276    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt" event={"ID":"84821dee-7355-48ea-b90e-c2b8efb7ef34","Type":"ContainerDied","Data":"b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d"}
Oct 30 23:03:20.790362 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.790303    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt" event={"ID":"84821dee-7355-48ea-b90e-c2b8efb7ef34","Type":"ContainerDied","Data":"815bad73d6c5e12943c224cccf58226188b2fa849a77991fa89a7339f68228ff"}
Oct 30 23:03:20.790362 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.790356    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"
Oct 30 23:03:20.809351 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.809325    4581 scope.go:122] "RemoveContainer" containerID="1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8"
Oct 30 23:03:20.828865 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.828747    4581 scope.go:122] "RemoveContainer" containerID="b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a"
Oct 30 23:03:20.829207 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:20.829174    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a\": container with ID starting with b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a not found: ID does not exist" containerID="b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a"
Oct 30 23:03:20.829335 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.829236    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a"} err="failed to get container status \"b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a\": rpc error: code = NotFound desc = could not find container \"b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a\": container with ID starting with b71aecc2dc97219433cfcdafe8900f95b793c92bd6739885c1cb82494789625a not found: ID does not exist"
Oct 30 23:03:20.829335 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.829261    4581 scope.go:122] "RemoveContainer" containerID="1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8"
Oct 30 23:03:20.829872 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:20.829720    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8\": container with ID starting with 1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8 not found: ID does not exist" containerID="1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8"
Oct 30 23:03:20.829872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.829754    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8"} err="failed to get container status \"1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8\": rpc error: code = NotFound desc = could not find container \"1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8\": container with ID starting with 1e852c8581843de6d18c78c07fc963cdaf16f2d692aab3e90816b134292898f8 not found: ID does not exist"
Oct 30 23:03:20.829872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.829778    4581 scope.go:122] "RemoveContainer" containerID="2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b"
Oct 30 23:03:20.836825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.836776    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"]
Oct 30 23:03:20.842192 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.842171    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d9f1c2-openshift-default-55dd7d9447-l89ql"]
Oct 30 23:03:20.853290 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.853265    4581 scope.go:122] "RemoveContainer" containerID="2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b"
Oct 30 23:03:20.853625 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:20.853605    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b\": container with ID starting with 2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b not found: ID does not exist" containerID="2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b"
Oct 30 23:03:20.853718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.853632    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b"} err="failed to get container status \"2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b\": rpc error: code = NotFound desc = could not find container \"2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b\": container with ID starting with 2413ee7f74d2bd9509779446747bd70d45286b8cfe21367b8896dc3284e34b2b not found: ID does not exist"
Oct 30 23:03:20.853718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.853649    4581 scope.go:122] "RemoveContainer" containerID="b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d"
Oct 30 23:03:20.867468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.867444    4581 scope.go:122] "RemoveContainer" containerID="b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d"
Oct 30 23:03:20.867589 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.867447    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.867589 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.867534    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.867589 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.867549    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4b4443-3f69-43a0-9a55-118b93e68702-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.867589 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.867564    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7kv2\" (UniqueName: \"kubernetes.io/projected/ae4b4443-3f69-43a0-9a55-118b93e68702-kube-api-access-v7kv2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.867827 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:20.867803    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d\": container with ID starting with b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d not found: ID does not exist" containerID="b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d"
Oct 30 23:03:20.867871 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.867837    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d"} err="failed to get container status \"b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d\": rpc error: code = NotFound desc = could not find container \"b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d\": container with ID starting with b0ed74fce16b93c1e8717010f1ec6aa1227c8250185c588b5c9e24344384f23d not found: ID does not exist"
Oct 30 23:03:20.899870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.899842    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"]
Oct 30 23:03:20.917661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.917628    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76d54a-84746b7f6b-br8pt"]
Oct 30 23:03:20.929872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.929843    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"]
Oct 30 23:03:20.934672 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:20.934645    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76d54a-fcfc75c9f-lvgx2"]
Oct 30 23:03:22.100508 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.100477    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa77793-4b17-4cc7-8731-fcfcfc0cec75" path="/var/lib/kubelet/pods/0aa77793-4b17-4cc7-8731-fcfcfc0cec75/volumes"
Oct 30 23:03:22.100981 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.100965    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84821dee-7355-48ea-b90e-c2b8efb7ef34" path="/var/lib/kubelet/pods/84821dee-7355-48ea-b90e-c2b8efb7ef34/volumes"
Oct 30 23:03:22.101442 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.101424    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2582fac-e117-49ec-8399-5c3a5a381773" path="/var/lib/kubelet/pods/a2582fac-e117-49ec-8399-5c3a5a381773/volumes"
Oct 30 23:03:22.102269 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.102249    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" path="/var/lib/kubelet/pods/ae4b4443-3f69-43a0-9a55-118b93e68702/volumes"
Oct 30 23:03:22.258009 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.257970    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"]
Oct 30 23:03:22.258760 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258737    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy"
Oct 30 23:03:22.258760 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258758    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258777    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258784    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258799    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258806    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258828    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258835    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258852    4581 cpu_manager.go:414] "containerMap: removing container" podUID="21135552-aa62-4db2-a43b-f9480fbbc6bd" containerName="revoke-keys"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258859    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21135552-aa62-4db2-a43b-f9480fbbc6bd" containerName="revoke-keys"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258881    4581 cpu_manager.go:414] "containerMap: removing container" podUID="84821dee-7355-48ea-b90e-c2b8efb7ef34" containerName="payload-processing"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258888    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84821dee-7355-48ea-b90e-c2b8efb7ef34" containerName="payload-processing"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258911    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0aa77793-4b17-4cc7-8731-fcfcfc0cec75" containerName="payload-pre-processing"
Oct 30 23:03:22.258955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.258919    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa77793-4b17-4cc7-8731-fcfcfc0cec75" containerName="payload-pre-processing"
Oct 30 23:03:22.259599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.259071    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy"
Oct 30 23:03:22.259599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.259088    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy"
Oct 30 23:03:22.259599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.259102    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="21135552-aa62-4db2-a43b-f9480fbbc6bd" containerName="revoke-keys"
Oct 30 23:03:22.259599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.259112    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="84821dee-7355-48ea-b90e-c2b8efb7ef34" containerName="payload-processing"
Oct 30 23:03:22.259599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.259124    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa77793-4b17-4cc7-8731-fcfcfc0cec75" containerName="payload-pre-processing"
Oct 30 23:03:22.264562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.264542    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.282571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.282538    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"]
Oct 30 23:03:22.286834 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.286809    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:22.286976 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.286904    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:03:22.345620 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.345594    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:22.345804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.345663    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:22.387252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.387194    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.387438 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.387313    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58sw6\" (UniqueName: \"kubernetes.io/projected/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-kube-api-access-58sw6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.387438 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.387382    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.488663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.488626    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.488835 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.488669    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58sw6\" (UniqueName: \"kubernetes.io/projected/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-kube-api-access-58sw6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.488835 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.488741    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.489466 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.489432    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.491311 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.491288    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.509420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.509387    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58sw6\" (UniqueName: \"kubernetes.io/projected/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-kube-api-access-58sw6\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.583190 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.583154    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:22.891409 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.891373    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"]
Oct 30 23:03:22.895494 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.895467    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-510f80-zzzj2"]
Oct 30 23:03:22.941606 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:22.941578    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"]
Oct 30 23:03:22.942726 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:22.942704    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdd9617_53e8_4fda_ba8f_6eaa4645ba85.slice/crio-c67d36751f101080efa39817d35a37328b6477edaa4c46164db42b4e975fbeb7 WatchSource:0}: Error finding container c67d36751f101080efa39817d35a37328b6477edaa4c46164db42b4e975fbeb7: Status 404 returned error can't find the container with id c67d36751f101080efa39817d35a37328b6477edaa4c46164db42b4e975fbeb7
Oct 30 23:03:23.011992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:23.011959    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"]
Oct 30 23:03:23.829384 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:23.829350    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9" event={"ID":"afdd9617-53e8-4fda-ba8f-6eaa4645ba85","Type":"ContainerStarted","Data":"12e70737ba9ee7187afa49686441f2927d9354feaa2cf83be85ef3d3e721dcd1"}
Oct 30 23:03:23.829818 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:23.829413    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9" event={"ID":"afdd9617-53e8-4fda-ba8f-6eaa4645ba85","Type":"ContainerStarted","Data":"c67d36751f101080efa39817d35a37328b6477edaa4c46164db42b4e975fbeb7"}
Oct 30 23:03:23.866855 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:23.866801    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9" podStartSLOduration=1.866787566 podStartE2EDuration="1.866787566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:23.863888815 +0000 UTC m=+3387.902827701" watchObservedRunningTime="2026-10-30 23:03:23.866787566 +0000 UTC m=+3387.905726470"
Oct 30 23:03:24.097822 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:24.097754    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4563a2d-4edd-40e7-a786-3fdcde95e0f5" path="/var/lib/kubelet/pods/d4563a2d-4edd-40e7-a786-3fdcde95e0f5/volumes"
Oct 30 23:03:24.854663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:24.854625    4581 generic.go:356] "Generic (PLEG): container finished" podID="afdd9617-53e8-4fda-ba8f-6eaa4645ba85" containerID="12e70737ba9ee7187afa49686441f2927d9354feaa2cf83be85ef3d3e721dcd1" exitCode=0
Oct 30 23:03:24.855139 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:24.854703    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9" event={"ID":"afdd9617-53e8-4fda-ba8f-6eaa4645ba85","Type":"ContainerDied","Data":"12e70737ba9ee7187afa49686441f2927d9354feaa2cf83be85ef3d3e721dcd1"}
Oct 30 23:03:26.037904 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.037876    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:26.148653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.148565    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-xks-ca\") pod \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") "
Oct 30 23:03:26.148653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.148647    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-openshift-ca\") pod \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") "
Oct 30 23:03:26.148898 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.148718    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-kube-api-access-58sw6\" (UniqueName: \"kubernetes.io/projected/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-kube-api-access-58sw6\") pod \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\" (UID: \"afdd9617-53e8-4fda-ba8f-6eaa4645ba85\") "
Oct 30 23:03:26.149324 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.149267    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-openshift-ca" pod "afdd9617-53e8-4fda-ba8f-6eaa4645ba85" (UID: "afdd9617-53e8-4fda-ba8f-6eaa4645ba85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:26.155863 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.155831    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-kube-api-access-58sw6" pod "afdd9617-53e8-4fda-ba8f-6eaa4645ba85" (UID: "afdd9617-53e8-4fda-ba8f-6eaa4645ba85"). InnerVolumeSpecName "kube-api-access-58sw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:26.156023 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.155967    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-xks-ca" pod "afdd9617-53e8-4fda-ba8f-6eaa4645ba85" (UID: "afdd9617-53e8-4fda-ba8f-6eaa4645ba85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:26.250550 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.250413    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:26.250550 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.250544    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58sw6\" (UniqueName: \"kubernetes.io/projected/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-kube-api-access-58sw6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:26.250550 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.250558    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd9617-53e8-4fda-ba8f-6eaa4645ba85-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:26.878539 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.878508    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"
Oct 30 23:03:26.878539 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.878524    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9" event={"ID":"afdd9617-53e8-4fda-ba8f-6eaa4645ba85","Type":"ContainerDied","Data":"c67d36751f101080efa39817d35a37328b6477edaa4c46164db42b4e975fbeb7"}
Oct 30 23:03:26.878814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:26.878566    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c67d36751f101080efa39817d35a37328b6477edaa4c46164db42b4e975fbeb7"
Oct 30 23:03:28.012394 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.012333    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg" podUID="6592319f-123c-4546-8a47-668a8fb03851" containerName="payload-processing" containerID="cri-o://6baeb8ca5cd1ad6b028ab5146ebc5d04e931a2eb73aad08b8b92f8e61ca62d67" gracePeriod=25
Oct 30 23:03:28.029137 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.029097    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"]
Oct 30 23:03:28.039164 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.039141    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0605cb02-zbgrj"]
Oct 30 23:03:28.108366 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.108331    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08507474-f98e-45eb-a390-a6aa9f21b39e" path="/var/lib/kubelet/pods/08507474-f98e-45eb-a390-a6aa9f21b39e/volumes"
Oct 30 23:03:28.764963 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.764932    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"]
Oct 30 23:03:28.765954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.765928    4581 cpu_manager.go:414] "containerMap: removing container" podUID="afdd9617-53e8-4fda-ba8f-6eaa4645ba85" containerName="revoke-keys"
Oct 30 23:03:28.765954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.765952    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdd9617-53e8-4fda-ba8f-6eaa4645ba85" containerName="revoke-keys"
Oct 30 23:03:28.766191 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.766178    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdd9617-53e8-4fda-ba8f-6eaa4645ba85" containerName="revoke-keys"
Oct 30 23:03:28.766259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.766193    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c069b4a9-3dd5-4144-a15e-e3df78c5f554" containerName="istio-proxy"
Oct 30 23:03:28.766259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.766207    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4b4443-3f69-43a0-9a55-118b93e68702" containerName="istio-proxy"
Oct 30 23:03:28.770919 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.770898    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:28.788370 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.788337    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"]
Oct 30 23:03:28.791647 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.791621    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:28.791783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.791714    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:28.805778 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.805726    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:28.805778 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.805746    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:03:28.916311 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.916275    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmq2\" (UniqueName: \"kubernetes.io/projected/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-kube-api-access-4dmq2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:28.916534 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.916468    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:28.916632 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:28.916617    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:29.017931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.017842    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmq2\" (UniqueName: \"kubernetes.io/projected/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-kube-api-access-4dmq2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:29.018389 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.017946    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:29.018389 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.018035    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:29.018633 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.018614    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:29.020436 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.020411    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:29.037876 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.037852    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmq2\" (UniqueName: \"kubernetes.io/projected/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-kube-api-access-4dmq2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:29.091443 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.091417    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:29.480109 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.480074    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"]
Oct 30 23:03:29.481976 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:29.481944    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41681e47_cd0e_4dc7_b0f2_e221441f5dd9.slice/crio-913bdb73bbf2ecfb05102743b7370fb8b129b8966c7dab0dfacbb8732e033136 WatchSource:0}: Error finding container 913bdb73bbf2ecfb05102743b7370fb8b129b8966c7dab0dfacbb8732e033136: Status 404 returned error can't find the container with id 913bdb73bbf2ecfb05102743b7370fb8b129b8966c7dab0dfacbb8732e033136
Oct 30 23:03:29.912262 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.910324    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd" event={"ID":"41681e47-cd0e-4dc7-b0f2-e221441f5dd9","Type":"ContainerStarted","Data":"7af6eb44c991d7b6d25b2faf8750da6f30ca26a3677580663a9f23233eaa634c"}
Oct 30 23:03:29.912262 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.910388    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd" event={"ID":"41681e47-cd0e-4dc7-b0f2-e221441f5dd9","Type":"ContainerStarted","Data":"913bdb73bbf2ecfb05102743b7370fb8b129b8966c7dab0dfacbb8732e033136"}
Oct 30 23:03:29.949896 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:29.949828    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd" podStartSLOduration=1.94981169 podStartE2EDuration="1.94981169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:29.946646317 +0000 UTC m=+3393.985585202" watchObservedRunningTime="2026-10-30 23:03:29.94981169 +0000 UTC m=+3393.988750575"
Oct 30 23:03:30.543593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:30.543558    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"]
Oct 30 23:03:30.544043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:30.543781    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy" containerID="cri-o://93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45" gracePeriod=30
Oct 30 23:03:30.920026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:30.919987    4581 generic.go:356] "Generic (PLEG): container finished" podID="41681e47-cd0e-4dc7-b0f2-e221441f5dd9" containerID="7af6eb44c991d7b6d25b2faf8750da6f30ca26a3677580663a9f23233eaa634c" exitCode=0
Oct 30 23:03:30.920260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:30.920075    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd" event={"ID":"41681e47-cd0e-4dc7-b0f2-e221441f5dd9","Type":"ContainerDied","Data":"7af6eb44c991d7b6d25b2faf8750da6f30ca26a3677580663a9f23233eaa634c"}
Oct 30 23:03:31.305131 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.305042    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"]
Oct 30 23:03:31.309578 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.309550    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.334149 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.334118    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"]
Oct 30 23:03:31.373807 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.373772    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.373993 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.373939    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rq8l\" (UniqueName: \"kubernetes.io/projected/2b31f3e8-f069-4538-8fd8-3550effd3903-kube-api-access-5rq8l\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.374121 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.374092    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.475855 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.475803    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.476051 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.475924    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.476105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.476084    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rq8l\" (UniqueName: \"kubernetes.io/projected/2b31f3e8-f069-4538-8fd8-3550effd3903-kube-api-access-5rq8l\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.476724 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.476694    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.478541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.478523    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.496297 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.496260    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rq8l\" (UniqueName: \"kubernetes.io/projected/2b31f3e8-f069-4538-8fd8-3550effd3903-kube-api-access-5rq8l\") pod \"maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.624961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.624929    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:31.941306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:31.941280    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"]
Oct 30 23:03:31.943266 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:31.943234    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b31f3e8_f069_4538_8fd8_3550effd3903.slice/crio-9b34d53ce5aba387c9e2519a9831632805c3d5ac014a0381f57a6fc06730bb60 WatchSource:0}: Error finding container 9b34d53ce5aba387c9e2519a9831632805c3d5ac014a0381f57a6fc06730bb60: Status 404 returned error can't find the container with id 9b34d53ce5aba387c9e2519a9831632805c3d5ac014a0381f57a6fc06730bb60
Oct 30 23:03:32.102358 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.102327    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:32.185775 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.185691    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-xks-ca\") pod \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") "
Oct 30 23:03:32.185939 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.185787    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-kube-api-access-4dmq2\" (UniqueName: \"kubernetes.io/projected/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-kube-api-access-4dmq2\") pod \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") "
Oct 30 23:03:32.185939 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.185845    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-openshift-ca\") pod \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\" (UID: \"41681e47-cd0e-4dc7-b0f2-e221441f5dd9\") "
Oct 30 23:03:32.186285 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.186241    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-openshift-ca" pod "41681e47-cd0e-4dc7-b0f2-e221441f5dd9" (UID: "41681e47-cd0e-4dc7-b0f2-e221441f5dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:32.186495 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.186471    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:32.188322 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.188294    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-xks-ca" pod "41681e47-cd0e-4dc7-b0f2-e221441f5dd9" (UID: "41681e47-cd0e-4dc7-b0f2-e221441f5dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:32.188587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.188564    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-kube-api-access-4dmq2" pod "41681e47-cd0e-4dc7-b0f2-e221441f5dd9" (UID: "41681e47-cd0e-4dc7-b0f2-e221441f5dd9"). InnerVolumeSpecName "kube-api-access-4dmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:32.287414 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.287369    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:32.287414 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.287401    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dmq2\" (UniqueName: \"kubernetes.io/projected/41681e47-cd0e-4dc7-b0f2-e221441f5dd9-kube-api-access-4dmq2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:32.942441 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.942403    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"
Oct 30 23:03:32.942441 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.942431    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd" event={"ID":"41681e47-cd0e-4dc7-b0f2-e221441f5dd9","Type":"ContainerDied","Data":"913bdb73bbf2ecfb05102743b7370fb8b129b8966c7dab0dfacbb8732e033136"}
Oct 30 23:03:32.942927 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.942484    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="913bdb73bbf2ecfb05102743b7370fb8b129b8966c7dab0dfacbb8732e033136"
Oct 30 23:03:32.945906 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.945195    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w" event={"ID":"2b31f3e8-f069-4538-8fd8-3550effd3903","Type":"ContainerStarted","Data":"f7b28e1231f7a1e1cc548fa75accd5dc27ae35fe6f1ef330068414ce6fc57029"}
Oct 30 23:03:32.945906 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.945317    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w" event={"ID":"2b31f3e8-f069-4538-8fd8-3550effd3903","Type":"ContainerStarted","Data":"9b34d53ce5aba387c9e2519a9831632805c3d5ac014a0381f57a6fc06730bb60"}
Oct 30 23:03:32.997018 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:32.996950    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w" podStartSLOduration=1.9969327 podStartE2EDuration="1.9969327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:32.99060232 +0000 UTC m=+3397.029541206" watchObservedRunningTime="2026-10-30 23:03:32.9969327 +0000 UTC m=+3397.035871587"
Oct 30 23:03:33.036101 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:33.036065    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg" podUID="6592319f-123c-4546-8a47-668a8fb03851" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:33.958428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:33.958390    4581 generic.go:356] "Generic (PLEG): container finished" podID="2b31f3e8-f069-4538-8fd8-3550effd3903" containerID="f7b28e1231f7a1e1cc548fa75accd5dc27ae35fe6f1ef330068414ce6fc57029" exitCode=0
Oct 30 23:03:33.958938 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:33.958430    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w" event={"ID":"2b31f3e8-f069-4538-8fd8-3550effd3903","Type":"ContainerDied","Data":"f7b28e1231f7a1e1cc548fa75accd5dc27ae35fe6f1ef330068414ce6fc57029"}
Oct 30 23:03:35.154788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.154757    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:35.227801 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.227708    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-xks-ca\") pod \"2b31f3e8-f069-4538-8fd8-3550effd3903\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") "
Oct 30 23:03:35.228000 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.227815    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b31f3e8-f069-4538-8fd8-3550effd3903-kube-api-access-5rq8l\" (UniqueName: \"kubernetes.io/projected/2b31f3e8-f069-4538-8fd8-3550effd3903-kube-api-access-5rq8l\") pod \"2b31f3e8-f069-4538-8fd8-3550effd3903\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") "
Oct 30 23:03:35.228079 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.228036    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-openshift-ca\") pod \"2b31f3e8-f069-4538-8fd8-3550effd3903\" (UID: \"2b31f3e8-f069-4538-8fd8-3550effd3903\") "
Oct 30 23:03:35.228435 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.228409    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-openshift-ca" pod "2b31f3e8-f069-4538-8fd8-3550effd3903" (UID: "2b31f3e8-f069-4538-8fd8-3550effd3903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:35.228725 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.228594    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:35.230207 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.230184    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b31f3e8-f069-4538-8fd8-3550effd3903-kube-api-access-5rq8l" pod "2b31f3e8-f069-4538-8fd8-3550effd3903" (UID: "2b31f3e8-f069-4538-8fd8-3550effd3903"). InnerVolumeSpecName "kube-api-access-5rq8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:35.230557 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.230534    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-xks-ca" pod "2b31f3e8-f069-4538-8fd8-3550effd3903" (UID: "2b31f3e8-f069-4538-8fd8-3550effd3903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:35.329873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.329838    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b31f3e8-f069-4538-8fd8-3550effd3903-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:35.329873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.329871    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rq8l\" (UniqueName: \"kubernetes.io/projected/2b31f3e8-f069-4538-8fd8-3550effd3903-kube-api-access-5rq8l\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:35.717581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.717527    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main" containerID="cri-o://5a87c4bbabb5b2f3df50a208c453c3c981a5e6c494fc4d75ff01fe3967ac4925" gracePeriod=45
Oct 30 23:03:35.745939 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.737143    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main" containerID="cri-o://a36607d55aebe1e7169391ccac3d40419441f5738a7c00719ff64cef0b328398" gracePeriod=45
Oct 30 23:03:35.853296 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.853265    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.937666    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-credential-socket\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.937768    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-podinfo\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.937790    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-token\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-token\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.937840    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-kube-api-access-fh42t\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-kube-api-access-fh42t\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.937867    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-certs\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.937937    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-data\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.937976    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-envoy\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.938015    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istiod-ca-cert\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.938581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.938072    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-socket\") pod \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\" (UID: \"9c0e492e-4281-4be3-bed8-8f57b2f13eb5\") "
Oct 30 23:03:35.939469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.939180    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-credential-socket" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.939469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.939364    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-socket" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.940044 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.939979    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-certs" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.940248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.940185    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-data" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.940348 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.940327    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istiod-ca-cert" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:35.976792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.976710    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-envoy" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:35.977254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.977190    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-token" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:35.977498 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.977468    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-podinfo" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:35.985758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.985695    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-kube-api-access-fh42t" pod "9c0e492e-4281-4be3-bed8-8f57b2f13eb5" (UID: "9c0e492e-4281-4be3-bed8-8f57b2f13eb5"). InnerVolumeSpecName "kube-api-access-fh42t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:35.989135 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.989094    4581 generic.go:356] "Generic (PLEG): container finished" podID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerID="5a87c4bbabb5b2f3df50a208c453c3c981a5e6c494fc4d75ff01fe3967ac4925" exitCode=0
Oct 30 23:03:35.989578 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.989288    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" event={"ID":"bbbf0e91-e65c-4c67-bd21-5274367f24af","Type":"ContainerDied","Data":"5a87c4bbabb5b2f3df50a208c453c3c981a5e6c494fc4d75ff01fe3967ac4925"}
Oct 30 23:03:35.992280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.992034    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"
Oct 30 23:03:35.992380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.992292    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w" event={"ID":"2b31f3e8-f069-4538-8fd8-3550effd3903","Type":"ContainerDied","Data":"9b34d53ce5aba387c9e2519a9831632805c3d5ac014a0381f57a6fc06730bb60"}
Oct 30 23:03:35.992380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:35.992361    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b34d53ce5aba387c9e2519a9831632805c3d5ac014a0381f57a6fc06730bb60"
Oct 30 23:03:36.014667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.011389    4581 generic.go:356] "Generic (PLEG): container finished" podID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerID="93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45" exitCode=0
Oct 30 23:03:36.014667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.011548    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" event={"ID":"9c0e492e-4281-4be3-bed8-8f57b2f13eb5","Type":"ContainerDied","Data":"93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45"}
Oct 30 23:03:36.014667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.011606    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx" event={"ID":"9c0e492e-4281-4be3-bed8-8f57b2f13eb5","Type":"ContainerDied","Data":"5ca8309fd3e5bb72896c9088ee7896cecac6b57fe625565b2035c1b9deb37aad"}
Oct 30 23:03:36.014667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.011641    4581 scope.go:122] "RemoveContainer" containerID="93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45"
Oct 30 23:03:36.014667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.011817    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"
Oct 30 23:03:36.014667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.013272    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"]
Oct 30 23:03:36.023115 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.023085    4581 generic.go:356] "Generic (PLEG): container finished" podID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerID="a36607d55aebe1e7169391ccac3d40419441f5738a7c00719ff64cef0b328398" exitCode=0
Oct 30 23:03:36.023310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.023191    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" event={"ID":"058e0a99-1b6c-4077-92a6-6182d6c63150","Type":"ContainerDied","Data":"a36607d55aebe1e7169391ccac3d40419441f5738a7c00719ff64cef0b328398"}
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.054011    4581 scope.go:122] "RemoveContainer" containerID="ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be"
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055143    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055168    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055185    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055200    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055241    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055256    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055269    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh42t\" (UniqueName: \"kubernetes.io/projected/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-kube-api-access-fh42t\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055284    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.057590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.055295    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0e492e-4281-4be3-bed8-8f57b2f13eb5-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.150190 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.150073    4581 scope.go:122] "RemoveContainer" containerID="93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45"
Oct 30 23:03:36.160249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.151680    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"]
Oct 30 23:03:36.169194 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:36.161140    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45\": container with ID starting with 93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45 not found: ID does not exist" containerID="93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45"
Oct 30 23:03:36.169194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.161190    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45"} err="failed to get container status \"93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45\": rpc error: code = NotFound desc = could not find container \"93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45\": container with ID starting with 93b2b8b4dea58c24280ac44f98ca2d94fd7056a2132e8e9f9e0f66596540de45 not found: ID does not exist"
Oct 30 23:03:36.169194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.161243    4581 scope.go:122] "RemoveContainer" containerID="ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be"
Oct 30 23:03:36.171098 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:36.170145    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be\": container with ID starting with ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be not found: ID does not exist" containerID="ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be"
Oct 30 23:03:36.171098 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.170205    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be"} err="failed to get container status \"ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be\": rpc error: code = NotFound desc = could not find container \"ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be\": container with ID starting with ac9495de31f1f9e04ac5fd8aefe7e565c620963e914888e915857a01afd8b1be not found: ID does not exist"
Oct 30 23:03:36.212681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.212617    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 23:03:36.214417 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.214393    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 23:03:36.258731 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.258686    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-tmp-dir\") pod \"bbbf0e91-e65c-4c67-bd21-5274367f24af\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") "
Oct 30 23:03:36.259039 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.259021    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-dshm\") pod \"058e0a99-1b6c-4077-92a6-6182d6c63150\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") "
Oct 30 23:03:36.259210 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.259195    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-tmp-dir\") pod \"058e0a99-1b6c-4077-92a6-6182d6c63150\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") "
Oct 30 23:03:36.259511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.259492    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbf0e91-e65c-4c67-bd21-5274367f24af-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf0e91-e65c-4c67-bd21-5274367f24af-tls-certs\") pod \"bbbf0e91-e65c-4c67-bd21-5274367f24af\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") "
Oct 30 23:03:36.259703 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.259676    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbf0e91-e65c-4c67-bd21-5274367f24af-kube-api-access-lsflv\" (UniqueName: \"kubernetes.io/projected/bbbf0e91-e65c-4c67-bd21-5274367f24af-kube-api-access-lsflv\") pod \"bbbf0e91-e65c-4c67-bd21-5274367f24af\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") "
Oct 30 23:03:36.260058 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.260031    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-home\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-home\") pod \"058e0a99-1b6c-4077-92a6-6182d6c63150\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") "
Oct 30 23:03:36.260415 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.260386    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058e0a99-1b6c-4077-92a6-6182d6c63150-kube-api-access-sgklh\" (UniqueName: \"kubernetes.io/projected/058e0a99-1b6c-4077-92a6-6182d6c63150-kube-api-access-sgklh\") pod \"058e0a99-1b6c-4077-92a6-6182d6c63150\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") "
Oct 30 23:03:36.260567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.260550    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-model-cache\") pod \"bbbf0e91-e65c-4c67-bd21-5274367f24af\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") "
Oct 30 23:03:36.260729 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.260714    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-model-cache\") pod \"058e0a99-1b6c-4077-92a6-6182d6c63150\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") "
Oct 30 23:03:36.261125 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.261107    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-home\") pod \"bbbf0e91-e65c-4c67-bd21-5274367f24af\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") "
Oct 30 23:03:36.263325 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.263301    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/058e0a99-1b6c-4077-92a6-6182d6c63150-tls-certs\" (UniqueName: \"kubernetes.io/secret/058e0a99-1b6c-4077-92a6-6182d6c63150-tls-certs\") pod \"058e0a99-1b6c-4077-92a6-6182d6c63150\" (UID: \"058e0a99-1b6c-4077-92a6-6182d6c63150\") "
Oct 30 23:03:36.263584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.263568    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-dshm\") pod \"bbbf0e91-e65c-4c67-bd21-5274367f24af\" (UID: \"bbbf0e91-e65c-4c67-bd21-5274367f24af\") "
Oct 30 23:03:36.264875 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.261545    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-home" pod "bbbf0e91-e65c-4c67-bd21-5274367f24af" (UID: "bbbf0e91-e65c-4c67-bd21-5274367f24af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.265387 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.265366    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-tmp-dir" pod "058e0a99-1b6c-4077-92a6-6182d6c63150" (UID: "058e0a99-1b6c-4077-92a6-6182d6c63150"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.266827 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.266492    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-tmp-dir" pod "bbbf0e91-e65c-4c67-bd21-5274367f24af" (UID: "bbbf0e91-e65c-4c67-bd21-5274367f24af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.268754 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.268727    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-model-cache" pod "058e0a99-1b6c-4077-92a6-6182d6c63150" (UID: "058e0a99-1b6c-4077-92a6-6182d6c63150"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.268944 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.268922    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-model-cache" pod "bbbf0e91-e65c-4c67-bd21-5274367f24af" (UID: "bbbf0e91-e65c-4c67-bd21-5274367f24af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.302288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.302103    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"]
Oct 30 23:03:36.325246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.324402    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-510f80-openshift-default-6465ccddcf-wjvgx"]
Oct 30 23:03:36.327047 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.326987    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-home" pod "058e0a99-1b6c-4077-92a6-6182d6c63150" (UID: "058e0a99-1b6c-4077-92a6-6182d6c63150"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.332538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.330648    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058e0a99-1b6c-4077-92a6-6182d6c63150-kube-api-access-sgklh" pod "058e0a99-1b6c-4077-92a6-6182d6c63150" (UID: "058e0a99-1b6c-4077-92a6-6182d6c63150"). InnerVolumeSpecName "kube-api-access-sgklh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:36.332538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.330753    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-dshm" pod "058e0a99-1b6c-4077-92a6-6182d6c63150" (UID: "058e0a99-1b6c-4077-92a6-6182d6c63150"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.332538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.330840    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-dshm" pod "bbbf0e91-e65c-4c67-bd21-5274367f24af" (UID: "bbbf0e91-e65c-4c67-bd21-5274367f24af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.332538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.330924    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbf0e91-e65c-4c67-bd21-5274367f24af-tls-certs" pod "bbbf0e91-e65c-4c67-bd21-5274367f24af" (UID: "bbbf0e91-e65c-4c67-bd21-5274367f24af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:36.332538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.331673    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf0e91-e65c-4c67-bd21-5274367f24af-kube-api-access-lsflv" pod "bbbf0e91-e65c-4c67-bd21-5274367f24af" (UID: "bbbf0e91-e65c-4c67-bd21-5274367f24af"). InnerVolumeSpecName "kube-api-access-lsflv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:36.334157 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.333777    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058e0a99-1b6c-4077-92a6-6182d6c63150-tls-certs" pod "058e0a99-1b6c-4077-92a6-6182d6c63150" (UID: "058e0a99-1b6c-4077-92a6-6182d6c63150"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:36.368020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.367986    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.368290 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.368270    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.368419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.368407    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369727    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369753    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf0e91-e65c-4c67-bd21-5274367f24af-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369768    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsflv\" (UniqueName: \"kubernetes.io/projected/bbbf0e91-e65c-4c67-bd21-5274367f24af-kube-api-access-lsflv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369782    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369799    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgklh\" (UniqueName: \"kubernetes.io/projected/058e0a99-1b6c-4077-92a6-6182d6c63150-kube-api-access-sgklh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369812    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369826    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e0a99-1b6c-4077-92a6-6182d6c63150-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369838    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0e91-e65c-4c67-bd21-5274367f24af-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.369886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.369850    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e0a99-1b6c-4077-92a6-6182d6c63150-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.766030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.765994    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"]
Oct 30 23:03:36.771853 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:36.771810    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-80ecb3-f7knk"]
Oct 30 23:03:37.057615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.057525    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"
Oct 30 23:03:37.057615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.057549    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d" event={"ID":"bbbf0e91-e65c-4c67-bd21-5274367f24af","Type":"ContainerDied","Data":"e2228442c605a3649c016f190fe9e1c53bbb97c3646597b75b98acf0622e231b"}
Oct 30 23:03:37.057615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.057600    4581 scope.go:122] "RemoveContainer" containerID="5a87c4bbabb5b2f3df50a208c453c3c981a5e6c494fc4d75ff01fe3967ac4925"
Oct 30 23:03:37.060970 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.060896    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"
Oct 30 23:03:37.060970 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.060920    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl" event={"ID":"058e0a99-1b6c-4077-92a6-6182d6c63150","Type":"ContainerDied","Data":"5d235d0ecdba9c16730b7f03e32cf18c93191c52a1a8e8b45c94e137cbd2bca5"}
Oct 30 23:03:37.073840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.073811    4581 scope.go:122] "RemoveContainer" containerID="a36607d55aebe1e7169391ccac3d40419441f5738a7c00719ff64cef0b328398"
Oct 30 23:03:37.118507 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.118472    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"]
Oct 30 23:03:37.122829 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.122804    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-w6-012ba0-kserve-85d876d77c-h977d"]
Oct 30 23:03:37.148984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.148943    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"]
Oct 30 23:03:37.155161 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:37.155124    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/simulator-w6-012ba0-kserve-d8f4d985f-97qgl"]
Oct 30 23:03:38.098567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:38.098536    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" path="/var/lib/kubelet/pods/058e0a99-1b6c-4077-92a6-6182d6c63150/volumes"
Oct 30 23:03:38.098987 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:38.098940    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986c2252-d28e-4d1f-a6ae-42954d85e70b" path="/var/lib/kubelet/pods/986c2252-d28e-4d1f-a6ae-42954d85e70b/volumes"
Oct 30 23:03:38.099403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:38.099387    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" path="/var/lib/kubelet/pods/9c0e492e-4281-4be3-bed8-8f57b2f13eb5/volumes"
Oct 30 23:03:38.100102 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:38.100087    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" path="/var/lib/kubelet/pods/bbbf0e91-e65c-4c67-bd21-5274367f24af/volumes"
Oct 30 23:03:40.817524 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:40.817487    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:03:41.711995 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:41.711962    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"]
Oct 30 23:03:41.743052 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:41.743016    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"]
Oct 30 23:03:43.036825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:43.036757    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg" podUID="6592319f-123c-4546-8a47-668a8fb03851" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:44.405036 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.404621    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"]
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405400    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405420    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405435    4581 cpu_manager.go:414] "containerMap: removing container" podUID="41681e47-cd0e-4dc7-b0f2-e221441f5dd9" containerName="revoke-keys"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405442    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41681e47-cd0e-4dc7-b0f2-e221441f5dd9" containerName="revoke-keys"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405455    4581 cpu_manager.go:414] "containerMap: removing container" podUID="2b31f3e8-f069-4538-8fd8-3550effd3903" containerName="revoke-keys"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405463    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b31f3e8-f069-4538-8fd8-3550effd3903" containerName="revoke-keys"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405496    4581 cpu_manager.go:414] "containerMap: removing container" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405504    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405531    4581 cpu_manager.go:414] "containerMap: removing container" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main"
Oct 30 23:03:44.405641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405538    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main"
Oct 30 23:03:44.406239 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405697    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbf0e91-e65c-4c67-bd21-5274367f24af" containerName="main"
Oct 30 23:03:44.406239 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405714    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="058e0a99-1b6c-4077-92a6-6182d6c63150" containerName="main"
Oct 30 23:03:44.406239 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405721    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy"
Oct 30 23:03:44.406239 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405731    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="41681e47-cd0e-4dc7-b0f2-e221441f5dd9" containerName="revoke-keys"
Oct 30 23:03:44.406239 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.405737    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b31f3e8-f069-4538-8fd8-3550effd3903" containerName="revoke-keys"
Oct 30 23:03:44.411241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.411203    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.431778 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.431748    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"]
Oct 30 23:03:44.433945 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.433915    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:44.434075 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.433981    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:44.460358 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.460328    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:44.460670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.460649    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:03:44.503425 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.503387    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpfb\" (UniqueName: \"kubernetes.io/projected/6ff10ba1-afd1-42d2-a842-264e22ae819e-kube-api-access-4gpfb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.503604 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.503439    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.503653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.503613    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.604797 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.604765    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.605029 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.604929    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.605100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.605075    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpfb\" (UniqueName: \"kubernetes.io/projected/6ff10ba1-afd1-42d2-a842-264e22ae819e-kube-api-access-4gpfb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.605537 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.605514    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.607471 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.607446    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.624154 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.624124    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpfb\" (UniqueName: \"kubernetes.io/projected/6ff10ba1-afd1-42d2-a842-264e22ae819e-kube-api-access-4gpfb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:44.727970 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:44.727889    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:45.061371 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:45.061331    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"]
Oct 30 23:03:45.063121 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:45.063089    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff10ba1_afd1_42d2_a842_264e22ae819e.slice/crio-0c4853c26f5f2141dad7ffa2a3b3acb8249229a590b50643014dbc28f73fd7ca WatchSource:0}: Error finding container 0c4853c26f5f2141dad7ffa2a3b3acb8249229a590b50643014dbc28f73fd7ca: Status 404 returned error can't find the container with id 0c4853c26f5f2141dad7ffa2a3b3acb8249229a590b50643014dbc28f73fd7ca
Oct 30 23:03:45.200323 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:45.200283    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"]
Oct 30 23:03:45.205861 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:45.205798    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5983ea-5hnjd"]
Oct 30 23:03:45.210695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:45.210658    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk" event={"ID":"6ff10ba1-afd1-42d2-a842-264e22ae819e","Type":"ContainerStarted","Data":"0c4853c26f5f2141dad7ffa2a3b3acb8249229a590b50643014dbc28f73fd7ca"}
Oct 30 23:03:46.101151 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:46.101120    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41681e47-cd0e-4dc7-b0f2-e221441f5dd9" path="/var/lib/kubelet/pods/41681e47-cd0e-4dc7-b0f2-e221441f5dd9/volumes"
Oct 30 23:03:46.221038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:46.221003    4581 generic.go:356] "Generic (PLEG): container finished" podID="6ff10ba1-afd1-42d2-a842-264e22ae819e" containerID="5dc1947500be6ee1d5e4b607f0ec78ff0695a2ee900d5611b42afd24e2cdfb81" exitCode=0
Oct 30 23:03:46.221245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:46.221092    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk" event={"ID":"6ff10ba1-afd1-42d2-a842-264e22ae819e","Type":"ContainerDied","Data":"5dc1947500be6ee1d5e4b607f0ec78ff0695a2ee900d5611b42afd24e2cdfb81"}
Oct 30 23:03:46.712806 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:46.712749    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc" podUID="9be81c22-942d-4049-b971-e55f040bdd92" containerName="payload-processing" containerID="cri-o://9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09" gracePeriod=25
Oct 30 23:03:46.743819 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:46.743785    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv" podUID="60eef6f3-d9eb-4716-bbb3-15f4e28249b8" containerName="payload-pre-processing" containerID="cri-o://0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0" gracePeriod=25
Oct 30 23:03:47.400438 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.400414    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:47.456758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.456720    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-xks-ca\") pod \"6ff10ba1-afd1-42d2-a842-264e22ae819e\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") "
Oct 30 23:03:47.456942 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.456818    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-openshift-ca\") pod \"6ff10ba1-afd1-42d2-a842-264e22ae819e\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") "
Oct 30 23:03:47.456942 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.456901    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff10ba1-afd1-42d2-a842-264e22ae819e-kube-api-access-4gpfb\" (UniqueName: \"kubernetes.io/projected/6ff10ba1-afd1-42d2-a842-264e22ae819e-kube-api-access-4gpfb\") pod \"6ff10ba1-afd1-42d2-a842-264e22ae819e\" (UID: \"6ff10ba1-afd1-42d2-a842-264e22ae819e\") "
Oct 30 23:03:47.457289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.457255    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-openshift-ca" pod "6ff10ba1-afd1-42d2-a842-264e22ae819e" (UID: "6ff10ba1-afd1-42d2-a842-264e22ae819e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:47.457522 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.457501    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:47.459832 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.459804    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-xks-ca" pod "6ff10ba1-afd1-42d2-a842-264e22ae819e" (UID: "6ff10ba1-afd1-42d2-a842-264e22ae819e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:47.461201 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.461177    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff10ba1-afd1-42d2-a842-264e22ae819e-kube-api-access-4gpfb" pod "6ff10ba1-afd1-42d2-a842-264e22ae819e" (UID: "6ff10ba1-afd1-42d2-a842-264e22ae819e"). InnerVolumeSpecName "kube-api-access-4gpfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:47.558552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.558463    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff10ba1-afd1-42d2-a842-264e22ae819e-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:47.558552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.558491    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gpfb\" (UniqueName: \"kubernetes.io/projected/6ff10ba1-afd1-42d2-a842-264e22ae819e-kube-api-access-4gpfb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:47.621004 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.618238    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"]
Oct 30 23:03:47.623259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:47.623174    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bed544-9xps9"]
Oct 30 23:03:48.098391 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.098357    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdd9617-53e8-4fda-ba8f-6eaa4645ba85" path="/var/lib/kubelet/pods/afdd9617-53e8-4fda-ba8f-6eaa4645ba85/volumes"
Oct 30 23:03:48.249550 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.249518    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"
Oct 30 23:03:48.249550 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.249527    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk" event={"ID":"6ff10ba1-afd1-42d2-a842-264e22ae819e","Type":"ContainerDied","Data":"0c4853c26f5f2141dad7ffa2a3b3acb8249229a590b50643014dbc28f73fd7ca"}
Oct 30 23:03:48.249796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.249582    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c4853c26f5f2141dad7ffa2a3b3acb8249229a590b50643014dbc28f73fd7ca"
Oct 30 23:03:48.255076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.255042    4581 generic.go:356] "Generic (PLEG): container finished" podID="6592319f-123c-4546-8a47-668a8fb03851" containerID="6baeb8ca5cd1ad6b028ab5146ebc5d04e931a2eb73aad08b8b92f8e61ca62d67" exitCode=0
Oct 30 23:03:48.255261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.255092    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg" event={"ID":"6592319f-123c-4546-8a47-668a8fb03851","Type":"ContainerDied","Data":"6baeb8ca5cd1ad6b028ab5146ebc5d04e931a2eb73aad08b8b92f8e61ca62d67"}
Oct 30 23:03:48.272933 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.272869    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 23:03:48.369826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.369793    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6592319f-123c-4546-8a47-668a8fb03851-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592319f-123c-4546-8a47-668a8fb03851-plugins-config-volume\") pod \"6592319f-123c-4546-8a47-668a8fb03851\" (UID: \"6592319f-123c-4546-8a47-668a8fb03851\") "
Oct 30 23:03:48.370034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.369949    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6592319f-123c-4546-8a47-668a8fb03851-kube-api-access-crqqv\" (UniqueName: \"kubernetes.io/projected/6592319f-123c-4546-8a47-668a8fb03851-kube-api-access-crqqv\") pod \"6592319f-123c-4546-8a47-668a8fb03851\" (UID: \"6592319f-123c-4546-8a47-668a8fb03851\") "
Oct 30 23:03:48.370302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.370271    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6592319f-123c-4546-8a47-668a8fb03851-plugins-config-volume" pod "6592319f-123c-4546-8a47-668a8fb03851" (UID: "6592319f-123c-4546-8a47-668a8fb03851"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:48.370616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.370596    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592319f-123c-4546-8a47-668a8fb03851-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:48.372961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.372925    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592319f-123c-4546-8a47-668a8fb03851-kube-api-access-crqqv" pod "6592319f-123c-4546-8a47-668a8fb03851" (UID: "6592319f-123c-4546-8a47-668a8fb03851"). InnerVolumeSpecName "kube-api-access-crqqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:48.472300 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:48.472266    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crqqv\" (UniqueName: \"kubernetes.io/projected/6592319f-123c-4546-8a47-668a8fb03851-kube-api-access-crqqv\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:49.135147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135111    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"]
Oct 30 23:03:49.135692 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135678    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6ff10ba1-afd1-42d2-a842-264e22ae819e" containerName="revoke-keys"
Oct 30 23:03:49.135741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135694    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff10ba1-afd1-42d2-a842-264e22ae819e" containerName="revoke-keys"
Oct 30 23:03:49.135741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135710    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy"
Oct 30 23:03:49.135741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135716    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy"
Oct 30 23:03:49.135845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135754    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6592319f-123c-4546-8a47-668a8fb03851" containerName="payload-processing"
Oct 30 23:03:49.135845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135761    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6592319f-123c-4546-8a47-668a8fb03851" containerName="payload-processing"
Oct 30 23:03:49.135907 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135844    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0e492e-4281-4be3-bed8-8f57b2f13eb5" containerName="istio-proxy"
Oct 30 23:03:49.135907 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135855    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6592319f-123c-4546-8a47-668a8fb03851" containerName="payload-processing"
Oct 30 23:03:49.135907 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.135862    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff10ba1-afd1-42d2-a842-264e22ae819e" containerName="revoke-keys"
Oct 30 23:03:49.139456 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.139441    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.156561 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.156505    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"]
Oct 30 23:03:49.161508 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.161487    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.162162 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.162137    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:49.162661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.162301    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:03:49.162661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.162377    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:49.164199 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.164171    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:49.177259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.171274    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"]
Oct 30 23:03:49.186259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.180827    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"]
Oct 30 23:03:49.267642 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.267613    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"
Oct 30 23:03:49.267857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.267610    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg" event={"ID":"6592319f-123c-4546-8a47-668a8fb03851","Type":"ContainerDied","Data":"9ba38c6fe72570a62d7c62b1a23f3afa73f37cc29ede898376efdd1027898072"}
Oct 30 23:03:49.267857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.267739    4581 scope.go:122] "RemoveContainer" containerID="6baeb8ca5cd1ad6b028ab5146ebc5d04e931a2eb73aad08b8b92f8e61ca62d67"
Oct 30 23:03:49.285515 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.285484    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7xjc\" (UniqueName: \"kubernetes.io/projected/87a7548e-51bd-4686-a413-a3f0e75306f6-kube-api-access-p7xjc\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.285668 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.285564    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.285668 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.285599    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.285789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.285771    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4fl\" (UniqueName: \"kubernetes.io/projected/62bb59d5-ee90-4095-8b52-c956cb3fff9f-kube-api-access-7l4fl\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.285847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.285831    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.285958 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.285935    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.318068 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.318037    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"]
Oct 30 23:03:49.322321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.322292    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-80ecb3-67699fb85f-7tctg"]
Oct 30 23:03:49.387662 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.387558    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7xjc\" (UniqueName: \"kubernetes.io/projected/87a7548e-51bd-4686-a413-a3f0e75306f6-kube-api-access-p7xjc\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.387662 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.387642    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.387925 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.387683    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.387925 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.387792    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4fl\" (UniqueName: \"kubernetes.io/projected/62bb59d5-ee90-4095-8b52-c956cb3fff9f-kube-api-access-7l4fl\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.387925 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.387836    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.387925 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.387883    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.388479 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.388453    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.388618 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.388591    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.390499 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.390477    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.390840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.390820    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.405344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.405314    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7xjc\" (UniqueName: \"kubernetes.io/projected/87a7548e-51bd-4686-a413-a3f0e75306f6-kube-api-access-p7xjc\") pod \"maas-api-revoke-sub-ac0ccdb8-mqk4k\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.407146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.407122    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4fl\" (UniqueName: \"kubernetes.io/projected/62bb59d5-ee90-4095-8b52-c956cb3fff9f-kube-api-access-7l4fl\") pod \"maas-api-revoke-sub-2446288b-94cfn\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.453794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.453762    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:49.485360 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.485330    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:49.766677 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.766645    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"]
Oct 30 23:03:49.768389 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:49.768356    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bb59d5_ee90_4095_8b52_c956cb3fff9f.slice/crio-0f5b4c9858e28c01cf8d0bb055d8113f5d891de5876b4750179b995e98b7d6d7 WatchSource:0}: Error finding container 0f5b4c9858e28c01cf8d0bb055d8113f5d891de5876b4750179b995e98b7d6d7: Status 404 returned error can't find the container with id 0f5b4c9858e28c01cf8d0bb055d8113f5d891de5876b4750179b995e98b7d6d7
Oct 30 23:03:49.820094 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:49.820060    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"]
Oct 30 23:03:49.821689 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:49.821664    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a7548e_51bd_4686_a413_a3f0e75306f6.slice/crio-9a1c3a1a83c43b954aa3ca17cf5320496a3d5c1f65f23d24fc6a310329407b7d WatchSource:0}: Error finding container 9a1c3a1a83c43b954aa3ca17cf5320496a3d5c1f65f23d24fc6a310329407b7d: Status 404 returned error can't find the container with id 9a1c3a1a83c43b954aa3ca17cf5320496a3d5c1f65f23d24fc6a310329407b7d
Oct 30 23:03:50.097184 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:50.097147    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6592319f-123c-4546-8a47-668a8fb03851" path="/var/lib/kubelet/pods/6592319f-123c-4546-8a47-668a8fb03851/volumes"
Oct 30 23:03:50.279543 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:50.279501    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn" event={"ID":"62bb59d5-ee90-4095-8b52-c956cb3fff9f","Type":"ContainerStarted","Data":"c0d0a646409845cdedec3ca777efa257404a599f0ac4e619638663cdd71de317"}
Oct 30 23:03:50.279721 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:50.279573    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn" event={"ID":"62bb59d5-ee90-4095-8b52-c956cb3fff9f","Type":"ContainerStarted","Data":"0f5b4c9858e28c01cf8d0bb055d8113f5d891de5876b4750179b995e98b7d6d7"}
Oct 30 23:03:50.281120 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:50.281096    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k" event={"ID":"87a7548e-51bd-4686-a413-a3f0e75306f6","Type":"ContainerStarted","Data":"0699dc731fbb12710f978fef08f906217bb0e274a59f5d38436dcbde069bb72d"}
Oct 30 23:03:50.281173 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:50.281144    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k" event={"ID":"87a7548e-51bd-4686-a413-a3f0e75306f6","Type":"ContainerStarted","Data":"9a1c3a1a83c43b954aa3ca17cf5320496a3d5c1f65f23d24fc6a310329407b7d"}
Oct 30 23:03:50.328334 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:50.325877    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn" podStartSLOduration=1.32585679 podStartE2EDuration="1.32585679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:50.31295992 +0000 UTC m=+3414.351898809" watchObservedRunningTime="2026-10-30 23:03:50.32585679 +0000 UTC m=+3414.364795677"
Oct 30 23:03:51.047595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.047496    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k" podStartSLOduration=2.047480649 podStartE2EDuration="2.047480649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:50.377449481 +0000 UTC m=+3414.416388369" watchObservedRunningTime="2026-10-30 23:03:51.047480649 +0000 UTC m=+3415.086419592"
Oct 30 23:03:51.051490 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.051463    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"]
Oct 30 23:03:51.055715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.055682    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main" containerID="cri-o://9d18f5ca0fd33b3b6b51b3f2f36df09d8c0e6ad627de06bc86149649067bb5ec" gracePeriod=45
Oct 30 23:03:51.067450 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.067425    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d4ba70-9sb77"]
Oct 30 23:03:51.224684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.224647    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main" containerID="cri-o://bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7" gracePeriod=45
Oct 30 23:03:51.301321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.295141    4581 generic.go:356] "Generic (PLEG): container finished" podID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerID="9d18f5ca0fd33b3b6b51b3f2f36df09d8c0e6ad627de06bc86149649067bb5ec" exitCode=0
Oct 30 23:03:51.301321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.295264    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" event={"ID":"9f861b1e-34e1-4f94-8074-f0cb46980b9c","Type":"ContainerDied","Data":"9d18f5ca0fd33b3b6b51b3f2f36df09d8c0e6ad627de06bc86149649067bb5ec"}
Oct 30 23:03:51.301321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.298255    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn" event={"ID":"62bb59d5-ee90-4095-8b52-c956cb3fff9f","Type":"ContainerDied","Data":"c0d0a646409845cdedec3ca777efa257404a599f0ac4e619638663cdd71de317"}
Oct 30 23:03:51.301321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.298271    4581 generic.go:356] "Generic (PLEG): container finished" podID="62bb59d5-ee90-4095-8b52-c956cb3fff9f" containerID="c0d0a646409845cdedec3ca777efa257404a599f0ac4e619638663cdd71de317" exitCode=0
Oct 30 23:03:51.301321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.300467    4581 generic.go:356] "Generic (PLEG): container finished" podID="87a7548e-51bd-4686-a413-a3f0e75306f6" containerID="0699dc731fbb12710f978fef08f906217bb0e274a59f5d38436dcbde069bb72d" exitCode=0
Oct 30 23:03:51.301321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.300508    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k" event={"ID":"87a7548e-51bd-4686-a413-a3f0e75306f6","Type":"ContainerDied","Data":"0699dc731fbb12710f978fef08f906217bb0e274a59f5d38436dcbde069bb72d"}
Oct 30 23:03:51.340470 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.340441    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 23:03:51.412765 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.412737    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tmp-dir\") pod \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") "
Oct 30 23:03:51.412944 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.412796    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-model-cache\") pod \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") "
Oct 30 23:03:51.412944 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.412837    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-home\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-home\") pod \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") "
Oct 30 23:03:51.412944 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.412896    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tls-certs\") pod \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") "
Oct 30 23:03:51.412944 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.412940    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-dshm\") pod \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") "
Oct 30 23:03:51.413162 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.412987    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f861b1e-34e1-4f94-8074-f0cb46980b9c-kube-api-access-xzc2g\" (UniqueName: \"kubernetes.io/projected/9f861b1e-34e1-4f94-8074-f0cb46980b9c-kube-api-access-xzc2g\") pod \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\" (UID: \"9f861b1e-34e1-4f94-8074-f0cb46980b9c\") "
Oct 30 23:03:51.413547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.413524    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-home" pod "9f861b1e-34e1-4f94-8074-f0cb46980b9c" (UID: "9f861b1e-34e1-4f94-8074-f0cb46980b9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.413705 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.413538    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tmp-dir" pod "9f861b1e-34e1-4f94-8074-f0cb46980b9c" (UID: "9f861b1e-34e1-4f94-8074-f0cb46980b9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.413705 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.413556    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-model-cache" pod "9f861b1e-34e1-4f94-8074-f0cb46980b9c" (UID: "9f861b1e-34e1-4f94-8074-f0cb46980b9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.422844 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.420088    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tls-certs" pod "9f861b1e-34e1-4f94-8074-f0cb46980b9c" (UID: "9f861b1e-34e1-4f94-8074-f0cb46980b9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:51.422844 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.420161    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f861b1e-34e1-4f94-8074-f0cb46980b9c-kube-api-access-xzc2g" pod "9f861b1e-34e1-4f94-8074-f0cb46980b9c" (UID: "9f861b1e-34e1-4f94-8074-f0cb46980b9c"). InnerVolumeSpecName "kube-api-access-xzc2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:51.434392 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.434339    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-dshm" pod "9f861b1e-34e1-4f94-8074-f0cb46980b9c" (UID: "9f861b1e-34e1-4f94-8074-f0cb46980b9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.521516 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.521465    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.521516 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.521504    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.521516 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.521520    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.522015 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.521534    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f861b1e-34e1-4f94-8074-f0cb46980b9c-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.522015 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.521553    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f861b1e-34e1-4f94-8074-f0cb46980b9c-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.522015 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.521570    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzc2g\" (UniqueName: \"kubernetes.io/projected/9f861b1e-34e1-4f94-8074-f0cb46980b9c-kube-api-access-xzc2g\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.531481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.531419    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 23:03:51.622493 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.622389    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-dshm\") pod \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") "
Oct 30 23:03:51.622493 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.622486    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-model-cache\") pod \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") "
Oct 30 23:03:51.622766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.622541    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-home\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-home\") pod \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") "
Oct 30 23:03:51.622766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.622619    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tls-certs\") pod \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") "
Oct 30 23:03:51.622766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.622714    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tmp-dir\") pod \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") "
Oct 30 23:03:51.622914 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.622783    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ac387d-d8d6-48f7-abb9-1583f893c4d0-kube-api-access-bgl55\" (UniqueName: \"kubernetes.io/projected/24ac387d-d8d6-48f7-abb9-1583f893c4d0-kube-api-access-bgl55\") pod \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\" (UID: \"24ac387d-d8d6-48f7-abb9-1583f893c4d0\") "
Oct 30 23:03:51.623063 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.623035    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-home" pod "24ac387d-d8d6-48f7-abb9-1583f893c4d0" (UID: "24ac387d-d8d6-48f7-abb9-1583f893c4d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.623336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.623306    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-model-cache" pod "24ac387d-d8d6-48f7-abb9-1583f893c4d0" (UID: "24ac387d-d8d6-48f7-abb9-1583f893c4d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.623449 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.623358    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.623720 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.623696    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tmp-dir" pod "24ac387d-d8d6-48f7-abb9-1583f893c4d0" (UID: "24ac387d-d8d6-48f7-abb9-1583f893c4d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.625428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.625405    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-dshm" pod "24ac387d-d8d6-48f7-abb9-1583f893c4d0" (UID: "24ac387d-d8d6-48f7-abb9-1583f893c4d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.626774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.625860    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ac387d-d8d6-48f7-abb9-1583f893c4d0-kube-api-access-bgl55" pod "24ac387d-d8d6-48f7-abb9-1583f893c4d0" (UID: "24ac387d-d8d6-48f7-abb9-1583f893c4d0"). InnerVolumeSpecName "kube-api-access-bgl55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:51.626774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.625866    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tls-certs" pod "24ac387d-d8d6-48f7-abb9-1583f893c4d0" (UID: "24ac387d-d8d6-48f7-abb9-1583f893c4d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:51.724995 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.724958    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.724995 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.724992    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.724995 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.725002    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.725279 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.725011    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgl55\" (UniqueName: \"kubernetes.io/projected/24ac387d-d8d6-48f7-abb9-1583f893c4d0-kube-api-access-bgl55\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.725279 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:51.725022    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ac387d-d8d6-48f7-abb9-1583f893c4d0-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.029793 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.029703    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"]
Oct 30 23:03:52.033940 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.033914    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48f25e1-rh5sj"]
Oct 30 23:03:52.103041 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.102997    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1099cf6e-bf69-4056-a065-4024c33622df" path="/var/lib/kubelet/pods/1099cf6e-bf69-4056-a065-4024c33622df/volumes"
Oct 30 23:03:52.103721 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.103700    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88cda865-1fb3-476f-be8c-8963729cf873" path="/var/lib/kubelet/pods/88cda865-1fb3-476f-be8c-8963729cf873/volumes"
Oct 30 23:03:52.311074 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.310968    4581 generic.go:356] "Generic (PLEG): container finished" podID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerID="bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7" exitCode=0
Oct 30 23:03:52.311074 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.311015    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"
Oct 30 23:03:52.311074 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.311061    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" event={"ID":"24ac387d-d8d6-48f7-abb9-1583f893c4d0","Type":"ContainerDied","Data":"bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7"}
Oct 30 23:03:52.311400 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.311106    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l" event={"ID":"24ac387d-d8d6-48f7-abb9-1583f893c4d0","Type":"ContainerDied","Data":"ed95758d8e5a6f114f033233541cffe656ebeded2bd888ad090d6faaabaf55a0"}
Oct 30 23:03:52.311400 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.311126    4581 scope.go:122] "RemoveContainer" containerID="bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7"
Oct 30 23:03:52.313065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.313043    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"
Oct 30 23:03:52.313186 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.313065    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs" event={"ID":"9f861b1e-34e1-4f94-8074-f0cb46980b9c","Type":"ContainerDied","Data":"49a40147359bc30b17dedb31d6837ae822c0a5b94e8b6974369ba63431d3b0ff"}
Oct 30 23:03:52.329656 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.329608    4581 scope.go:122] "RemoveContainer" containerID="bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7"
Oct 30 23:03:52.330110 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:52.330078    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7\": container with ID starting with bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7 not found: ID does not exist" containerID="bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7"
Oct 30 23:03:52.330288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.330122    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7"} err="failed to get container status \"bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7\": rpc error: code = NotFound desc = could not find container \"bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7\": container with ID starting with bf22d4ec1784c68a23824a8761a73aff8b532545a8b33a96df94f280a082a6f7 not found: ID does not exist"
Oct 30 23:03:52.330288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.330150    4581 scope.go:122] "RemoveContainer" containerID="9d18f5ca0fd33b3b6b51b3f2f36df09d8c0e6ad627de06bc86149649067bb5ec"
Oct 30 23:03:52.362477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.362193    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"]
Oct 30 23:03:52.371707 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.371674    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/distinct-2-w5-54114e-kserve-6899fc7795-65rhs"]
Oct 30 23:03:52.412766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.406824    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"]
Oct 30 23:03:52.413208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.412979    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/simulator-w5-54114e-kserve-7df5c4f9bd-flm8l"]
Oct 30 23:03:52.554583 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.554553    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:52.556805 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.556787    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:52.634748 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.634718    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a7548e-51bd-4686-a413-a3f0e75306f6-kube-api-access-p7xjc\" (UniqueName: \"kubernetes.io/projected/87a7548e-51bd-4686-a413-a3f0e75306f6-kube-api-access-p7xjc\") pod \"87a7548e-51bd-4686-a413-a3f0e75306f6\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") "
Oct 30 23:03:52.634931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.634783    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-xks-ca\") pod \"87a7548e-51bd-4686-a413-a3f0e75306f6\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") "
Oct 30 23:03:52.634931 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.634824    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-openshift-ca\") pod \"87a7548e-51bd-4686-a413-a3f0e75306f6\" (UID: \"87a7548e-51bd-4686-a413-a3f0e75306f6\") "
Oct 30 23:03:52.635044 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.634940    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62bb59d5-ee90-4095-8b52-c956cb3fff9f-kube-api-access-7l4fl\" (UniqueName: \"kubernetes.io/projected/62bb59d5-ee90-4095-8b52-c956cb3fff9f-kube-api-access-7l4fl\") pod \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") "
Oct 30 23:03:52.635044 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.635023    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-xks-ca\") pod \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") "
Oct 30 23:03:52.635170 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.635060    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-openshift-ca\") pod \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\" (UID: \"62bb59d5-ee90-4095-8b52-c956cb3fff9f\") "
Oct 30 23:03:52.635350 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.635321    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-openshift-ca" pod "87a7548e-51bd-4686-a413-a3f0e75306f6" (UID: "87a7548e-51bd-4686-a413-a3f0e75306f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:52.635696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.635659    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-openshift-ca" pod "62bb59d5-ee90-4095-8b52-c956cb3fff9f" (UID: "62bb59d5-ee90-4095-8b52-c956cb3fff9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:52.635696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.635672    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.638368 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.638344    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a7548e-51bd-4686-a413-a3f0e75306f6-kube-api-access-p7xjc" pod "87a7548e-51bd-4686-a413-a3f0e75306f6" (UID: "87a7548e-51bd-4686-a413-a3f0e75306f6"). InnerVolumeSpecName "kube-api-access-p7xjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:52.638758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.638732    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-xks-ca" pod "62bb59d5-ee90-4095-8b52-c956cb3fff9f" (UID: "62bb59d5-ee90-4095-8b52-c956cb3fff9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:52.638934 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.638913    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-xks-ca" pod "87a7548e-51bd-4686-a413-a3f0e75306f6" (UID: "87a7548e-51bd-4686-a413-a3f0e75306f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:52.639621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.639593    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bb59d5-ee90-4095-8b52-c956cb3fff9f-kube-api-access-7l4fl" pod "62bb59d5-ee90-4095-8b52-c956cb3fff9f" (UID: "62bb59d5-ee90-4095-8b52-c956cb3fff9f"). InnerVolumeSpecName "kube-api-access-7l4fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:52.736816 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.736774    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l4fl\" (UniqueName: \"kubernetes.io/projected/62bb59d5-ee90-4095-8b52-c956cb3fff9f-kube-api-access-7l4fl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.736816 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.736818    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.737089 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.736836    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bb59d5-ee90-4095-8b52-c956cb3fff9f-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.737089 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.736853    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7xjc\" (UniqueName: \"kubernetes.io/projected/87a7548e-51bd-4686-a413-a3f0e75306f6-kube-api-access-p7xjc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.737089 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.736870    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a7548e-51bd-4686-a413-a3f0e75306f6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.743288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.739921    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"]
Oct 30 23:03:52.743288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:52.740210    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy" containerID="cri-o://f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d" gracePeriod=30
Oct 30 23:03:53.323762 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.323732    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"
Oct 30 23:03:53.324287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.323760    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn" event={"ID":"62bb59d5-ee90-4095-8b52-c956cb3fff9f","Type":"ContainerDied","Data":"0f5b4c9858e28c01cf8d0bb055d8113f5d891de5876b4750179b995e98b7d6d7"}
Oct 30 23:03:53.324287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.323815    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5b4c9858e28c01cf8d0bb055d8113f5d891de5876b4750179b995e98b7d6d7"
Oct 30 23:03:53.325417 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.325399    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"
Oct 30 23:03:53.325540 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.325412    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k" event={"ID":"87a7548e-51bd-4686-a413-a3f0e75306f6","Type":"ContainerDied","Data":"9a1c3a1a83c43b954aa3ca17cf5320496a3d5c1f65f23d24fc6a310329407b7d"}
Oct 30 23:03:53.325540 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.325474    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a1c3a1a83c43b954aa3ca17cf5320496a3d5c1f65f23d24fc6a310329407b7d"
Oct 30 23:03:53.342365 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342341    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"]
Oct 30 23:03:53.342891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342879    4581 cpu_manager.go:414] "containerMap: removing container" podUID="87a7548e-51bd-4686-a413-a3f0e75306f6" containerName="revoke-keys"
Oct 30 23:03:53.342941 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342894    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a7548e-51bd-4686-a413-a3f0e75306f6" containerName="revoke-keys"
Oct 30 23:03:53.342941 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342915    4581 cpu_manager.go:414] "containerMap: removing container" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main"
Oct 30 23:03:53.342941 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342924    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main"
Oct 30 23:03:53.343039 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342968    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main"
Oct 30 23:03:53.343039 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342973    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main"
Oct 30 23:03:53.343039 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342981    4581 cpu_manager.go:414] "containerMap: removing container" podUID="62bb59d5-ee90-4095-8b52-c956cb3fff9f" containerName="revoke-keys"
Oct 30 23:03:53.343039 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.342987    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62bb59d5-ee90-4095-8b52-c956cb3fff9f" containerName="revoke-keys"
Oct 30 23:03:53.343208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.343081    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" containerName="main"
Oct 30 23:03:53.343208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.343090    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="62bb59d5-ee90-4095-8b52-c956cb3fff9f" containerName="revoke-keys"
Oct 30 23:03:53.343208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.343098    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a7548e-51bd-4686-a413-a3f0e75306f6" containerName="revoke-keys"
Oct 30 23:03:53.343208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.343106    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" containerName="main"
Oct 30 23:03:53.346877 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.346855    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.365821 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.365790    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:53.366297 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.365979    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:53.370206 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.370151    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"]
Oct 30 23:03:53.382981 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.382955    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:53.383845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.383823    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:03:53.447435 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.447402    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29hrm\" (UniqueName: \"kubernetes.io/projected/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-kube-api-access-29hrm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.447650 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.447440    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.447650 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.447634    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.548813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.548776    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29hrm\" (UniqueName: \"kubernetes.io/projected/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-kube-api-access-29hrm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.548813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.548815    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.549028 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.548911    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.549573 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.549548    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.551603 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.551573    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.566767 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.566736    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29hrm\" (UniqueName: \"kubernetes.io/projected/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-kube-api-access-29hrm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:53.663562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:53.663532    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:54.012459 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:54.012429    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"]
Oct 30 23:03:54.014810 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:54.014781    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aca8e4_2f2d_4801_a982_b99184ee2e8c.slice/crio-de0130ac62c75977563e55b6b3518694bb4f8eb9420d2c23ad6fbf6c49fea249 WatchSource:0}: Error finding container de0130ac62c75977563e55b6b3518694bb4f8eb9420d2c23ad6fbf6c49fea249: Status 404 returned error can't find the container with id de0130ac62c75977563e55b6b3518694bb4f8eb9420d2c23ad6fbf6c49fea249
Oct 30 23:03:54.096945 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:54.096910    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ac387d-d8d6-48f7-abb9-1583f893c4d0" path="/var/lib/kubelet/pods/24ac387d-d8d6-48f7-abb9-1583f893c4d0/volumes"
Oct 30 23:03:54.097410 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:54.097391    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f861b1e-34e1-4f94-8074-f0cb46980b9c" path="/var/lib/kubelet/pods/9f861b1e-34e1-4f94-8074-f0cb46980b9c/volumes"
Oct 30 23:03:54.340367 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:54.340324    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4" event={"ID":"b3aca8e4-2f2d-4801-a982-b99184ee2e8c","Type":"ContainerStarted","Data":"47f9cb25a32d13ec53238c8b92e4b4b9f594b12e0b48a97f9f2e96b8ab694d70"}
Oct 30 23:03:54.340880 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:54.340410    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4" event={"ID":"b3aca8e4-2f2d-4801-a982-b99184ee2e8c","Type":"ContainerStarted","Data":"de0130ac62c75977563e55b6b3518694bb4f8eb9420d2c23ad6fbf6c49fea249"}
Oct 30 23:03:54.378260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:54.378184    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4" podStartSLOduration=1.3781652420000001 podStartE2EDuration="1.378165242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:54.37363228 +0000 UTC m=+3418.412571165" watchObservedRunningTime="2026-10-30 23:03:54.378165242 +0000 UTC m=+3418.417104130"
Oct 30 23:03:55.300022 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.299923    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"]
Oct 30 23:03:55.300571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.300535    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy" containerID="cri-o://8c82898fbeee21507fcef8fff68b17d10099b7bdb5772551e282034624391627" gracePeriod=30
Oct 30 23:03:55.398564 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.398522    4581 generic.go:356] "Generic (PLEG): container finished" podID="b3aca8e4-2f2d-4801-a982-b99184ee2e8c" containerID="47f9cb25a32d13ec53238c8b92e4b4b9f594b12e0b48a97f9f2e96b8ab694d70" exitCode=0
Oct 30 23:03:55.400906 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.398989    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4" event={"ID":"b3aca8e4-2f2d-4801-a982-b99184ee2e8c","Type":"ContainerDied","Data":"47f9cb25a32d13ec53238c8b92e4b4b9f594b12e0b48a97f9f2e96b8ab694d70"}
Oct 30 23:03:55.881375 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.881343    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"]
Oct 30 23:03:55.885588 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.885568    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:55.914835 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.914803    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"]
Oct 30 23:03:55.985193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.985161    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4x7n\" (UniqueName: \"kubernetes.io/projected/087d76fb-5e85-4885-9bca-2c0f91e8dff7-kube-api-access-v4x7n\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:55.985402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.985255    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:55.985492 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:55.985396    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:56.087336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.087299    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:56.087543 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.087465    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4x7n\" (UniqueName: \"kubernetes.io/projected/087d76fb-5e85-4885-9bca-2c0f91e8dff7-kube-api-access-v4x7n\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:56.087543 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.087514    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:56.088133 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.088100    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:56.090699 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.090670    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:56.108511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.108484    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4x7n\" (UniqueName: \"kubernetes.io/projected/087d76fb-5e85-4885-9bca-2c0f91e8dff7-kube-api-access-v4x7n\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:56.212251 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.212156    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:56.523648 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.523602    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"]
Oct 30 23:03:56.525531 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:56.525498    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087d76fb_5e85_4885_9bca_2c0f91e8dff7.slice/crio-53f08a93bc964f67b29e8572c5a53bfac956f57188d7e9c08312a10b2add845c WatchSource:0}: Error finding container 53f08a93bc964f67b29e8572c5a53bfac956f57188d7e9c08312a10b2add845c: Status 404 returned error can't find the container with id 53f08a93bc964f67b29e8572c5a53bfac956f57188d7e9c08312a10b2add845c
Oct 30 23:03:56.542958 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.542921    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv" podUID="60eef6f3-d9eb-4716-bbb3-15f4e28249b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:56.543185 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.543157    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc" podUID="9be81c22-942d-4049-b971-e55f040bdd92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:03:56.572255 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.572208    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:56.695304 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.695268    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-openshift-ca\") pod \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") "
Oct 30 23:03:56.695514 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.695356    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-xks-ca\") pod \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") "
Oct 30 23:03:56.695599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.695513    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-kube-api-access-29hrm\" (UniqueName: \"kubernetes.io/projected/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-kube-api-access-29hrm\") pod \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\" (UID: \"b3aca8e4-2f2d-4801-a982-b99184ee2e8c\") "
Oct 30 23:03:56.695696 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.695667    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-openshift-ca" pod "b3aca8e4-2f2d-4801-a982-b99184ee2e8c" (UID: "b3aca8e4-2f2d-4801-a982-b99184ee2e8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:56.695970 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.695948    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:56.697668 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.697600    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-xks-ca" pod "b3aca8e4-2f2d-4801-a982-b99184ee2e8c" (UID: "b3aca8e4-2f2d-4801-a982-b99184ee2e8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:56.697745 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.697667    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-kube-api-access-29hrm" pod "b3aca8e4-2f2d-4801-a982-b99184ee2e8c" (UID: "b3aca8e4-2f2d-4801-a982-b99184ee2e8c"). InnerVolumeSpecName "kube-api-access-29hrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:56.797711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.797665    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29hrm\" (UniqueName: \"kubernetes.io/projected/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-kube-api-access-29hrm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:56.797711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:56.797704    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aca8e4-2f2d-4801-a982-b99184ee2e8c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:57.362242 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.354585    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"]
Oct 30 23:03:57.362242 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.355580    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b3aca8e4-2f2d-4801-a982-b99184ee2e8c" containerName="revoke-keys"
Oct 30 23:03:57.362242 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.355599    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3aca8e4-2f2d-4801-a982-b99184ee2e8c" containerName="revoke-keys"
Oct 30 23:03:57.362242 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.355744    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3aca8e4-2f2d-4801-a982-b99184ee2e8c" containerName="revoke-keys"
Oct 30 23:03:57.366701 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.366673    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.386098 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.385789    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"]
Oct 30 23:03:57.428004 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.427884    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx" event={"ID":"087d76fb-5e85-4885-9bca-2c0f91e8dff7","Type":"ContainerStarted","Data":"8047ec68bdc841786f7c9bfd1b647287b891054f729e762717ed9b0bfb02a53f"}
Oct 30 23:03:57.428004 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.427946    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx" event={"ID":"087d76fb-5e85-4885-9bca-2c0f91e8dff7","Type":"ContainerStarted","Data":"53f08a93bc964f67b29e8572c5a53bfac956f57188d7e9c08312a10b2add845c"}
Oct 30 23:03:57.429824 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.429792    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"
Oct 30 23:03:57.429824 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.429797    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4" event={"ID":"b3aca8e4-2f2d-4801-a982-b99184ee2e8c","Type":"ContainerDied","Data":"de0130ac62c75977563e55b6b3518694bb4f8eb9420d2c23ad6fbf6c49fea249"}
Oct 30 23:03:57.430030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.429831    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de0130ac62c75977563e55b6b3518694bb4f8eb9420d2c23ad6fbf6c49fea249"
Oct 30 23:03:57.469965 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.469917    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx" podStartSLOduration=2.469897077 podStartE2EDuration="2.469897077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:57.46914384 +0000 UTC m=+3421.508082726" watchObservedRunningTime="2026-10-30 23:03:57.469897077 +0000 UTC m=+3421.508835963"
Oct 30 23:03:57.506057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.506015    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.506266 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.506063    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wr6d\" (UniqueName: \"kubernetes.io/projected/4ea21218-4240-428e-879a-14a7d982a62c-kube-api-access-7wr6d\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.506266 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.506152    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.607110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.607080    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.607619 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.607125    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wr6d\" (UniqueName: \"kubernetes.io/projected/4ea21218-4240-428e-879a-14a7d982a62c-kube-api-access-7wr6d\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.607619 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.607153    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.607880 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.607859    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.609632 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.609612    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.624984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.624928    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wr6d\" (UniqueName: \"kubernetes.io/projected/4ea21218-4240-428e-879a-14a7d982a62c-kube-api-access-7wr6d\") pod \"maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:57.690530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:57.690493    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:03:58.008728 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.008538    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"]
Oct 30 23:03:58.012420 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:03:58.012394    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea21218_4240_428e_879a_14a7d982a62c.slice/crio-9b157cbf3294aa6bc7cf165094ec996ad55eb5e9c4b2545289e89e3b40fa0a2d WatchSource:0}: Error finding container 9b157cbf3294aa6bc7cf165094ec996ad55eb5e9c4b2545289e89e3b40fa0a2d: Status 404 returned error can't find the container with id 9b157cbf3294aa6bc7cf165094ec996ad55eb5e9c4b2545289e89e3b40fa0a2d
Oct 30 23:03:58.019339 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.019318    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 23:03:58.112344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112253    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istiod-ca-cert\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.112500 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112485    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-socket\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.112571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112531    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-certs\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.112641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112593    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istiod-ca-cert" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:58.112679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112647    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-credential-socket\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.112720 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112684    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-podinfo\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.112754 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112741    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-kube-api-access-glk4n\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-kube-api-access-glk4n\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.112809 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112773    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-data\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.112859 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112814    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-token\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.112859 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112824    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-socket" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.112859 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112833    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-certs" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.112965 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.112871    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-envoy\") pod \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\" (UID: \"998ef2e0-9da0-4c59-8d85-27fc27427cc3\") "
Oct 30 23:03:58.113102 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.113020    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-credential-socket" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.113371 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.113196    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-data" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.114193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.113860    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.114193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.113885    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.114193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.113901    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.114193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.113917    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.114193 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.113932    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.115692 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.115654    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-podinfo" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:58.116142 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.116118    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-token" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:58.116723 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.116697    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-kube-api-access-glk4n" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "kube-api-access-glk4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:58.125652 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.125615    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-envoy" pod "998ef2e0-9da0-4c59-8d85-27fc27427cc3" (UID: "998ef2e0-9da0-4c59-8d85-27fc27427cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:58.215771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.215679    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.215771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.215718    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glk4n\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-kube-api-access-glk4n\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.215771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.215733    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.215771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.215746    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998ef2e0-9da0-4c59-8d85-27fc27427cc3-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:58.440595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.440561    4581 generic.go:356] "Generic (PLEG): container finished" podID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerID="f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d" exitCode=0
Oct 30 23:03:58.440814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.440624    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"
Oct 30 23:03:58.440814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.440648    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" event={"ID":"998ef2e0-9da0-4c59-8d85-27fc27427cc3","Type":"ContainerDied","Data":"f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d"}
Oct 30 23:03:58.440814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.440713    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6" event={"ID":"998ef2e0-9da0-4c59-8d85-27fc27427cc3","Type":"ContainerDied","Data":"49d9188a3b287aba38b5466ae14afd94f1d002b11631ff33dcce232828e52c0e"}
Oct 30 23:03:58.440814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.440746    4581 scope.go:122] "RemoveContainer" containerID="f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d"
Oct 30 23:03:58.442711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.442679    4581 generic.go:356] "Generic (PLEG): container finished" podID="087d76fb-5e85-4885-9bca-2c0f91e8dff7" containerID="8047ec68bdc841786f7c9bfd1b647287b891054f729e762717ed9b0bfb02a53f" exitCode=0
Oct 30 23:03:58.442928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.442756    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx" event={"ID":"087d76fb-5e85-4885-9bca-2c0f91e8dff7","Type":"ContainerDied","Data":"8047ec68bdc841786f7c9bfd1b647287b891054f729e762717ed9b0bfb02a53f"}
Oct 30 23:03:58.445495 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.445177    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns" event={"ID":"4ea21218-4240-428e-879a-14a7d982a62c","Type":"ContainerStarted","Data":"577cff746e4e4ee11b55508107762694a1cc834fd972e2c25462f4425e2175c7"}
Oct 30 23:03:58.445495 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.445267    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns" event={"ID":"4ea21218-4240-428e-879a-14a7d982a62c","Type":"ContainerStarted","Data":"9b157cbf3294aa6bc7cf165094ec996ad55eb5e9c4b2545289e89e3b40fa0a2d"}
Oct 30 23:03:58.456748 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.456729    4581 scope.go:122] "RemoveContainer" containerID="f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d"
Oct 30 23:03:58.457052 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:03:58.457033    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d\": container with ID starting with f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d not found: ID does not exist" containerID="f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d"
Oct 30 23:03:58.457094 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.457061    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d"} err="failed to get container status \"f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d\": rpc error: code = NotFound desc = could not find container \"f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d\": container with ID starting with f240ed7582d025d2549f42524ef7464ee706d59f635fc2be52af8ba9c80ac45d not found: ID does not exist"
Oct 30 23:03:58.489426 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.489339    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"]
Oct 30 23:03:58.499262 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.499234    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5983ea-openshift-default-849df9ccdf-ffzk6"]
Oct 30 23:03:58.528514 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:58.528457    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns" podStartSLOduration=1.528439383 podStartE2EDuration="1.528439383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:58.525881021 +0000 UTC m=+3422.564819908" watchObservedRunningTime="2026-10-30 23:03:58.528439383 +0000 UTC m=+3422.567378270"
Oct 30 23:03:59.455571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.455541    4581 generic.go:356] "Generic (PLEG): container finished" podID="4ea21218-4240-428e-879a-14a7d982a62c" containerID="577cff746e4e4ee11b55508107762694a1cc834fd972e2c25462f4425e2175c7" exitCode=0
Oct 30 23:03:59.456007 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.455633    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns" event={"ID":"4ea21218-4240-428e-879a-14a7d982a62c","Type":"ContainerDied","Data":"577cff746e4e4ee11b55508107762694a1cc834fd972e2c25462f4425e2175c7"}
Oct 30 23:03:59.626096 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.626061    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:03:59.734245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.734101    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-openshift-ca\") pod \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") "
Oct 30 23:03:59.734245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.734180    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087d76fb-5e85-4885-9bca-2c0f91e8dff7-kube-api-access-v4x7n\" (UniqueName: \"kubernetes.io/projected/087d76fb-5e85-4885-9bca-2c0f91e8dff7-kube-api-access-v4x7n\") pod \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") "
Oct 30 23:03:59.734509 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.734418    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-xks-ca\") pod \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\" (UID: \"087d76fb-5e85-4885-9bca-2c0f91e8dff7\") "
Oct 30 23:03:59.734568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.734527    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-openshift-ca" pod "087d76fb-5e85-4885-9bca-2c0f91e8dff7" (UID: "087d76fb-5e85-4885-9bca-2c0f91e8dff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:59.735046 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.734944    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:59.736653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.736630    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087d76fb-5e85-4885-9bca-2c0f91e8dff7-kube-api-access-v4x7n" pod "087d76fb-5e85-4885-9bca-2c0f91e8dff7" (UID: "087d76fb-5e85-4885-9bca-2c0f91e8dff7"). InnerVolumeSpecName "kube-api-access-v4x7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:59.736809 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.736789    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-xks-ca" pod "087d76fb-5e85-4885-9bca-2c0f91e8dff7" (UID: "087d76fb-5e85-4885-9bca-2c0f91e8dff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:59.836485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.836447    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4x7n\" (UniqueName: \"kubernetes.io/projected/087d76fb-5e85-4885-9bca-2c0f91e8dff7-kube-api-access-v4x7n\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:03:59.836485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:03:59.836485    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087d76fb-5e85-4885-9bca-2c0f91e8dff7-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.098209 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.098125    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" path="/var/lib/kubelet/pods/998ef2e0-9da0-4c59-8d85-27fc27427cc3/volumes"
Oct 30 23:04:00.472735 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.472702    4581 generic.go:356] "Generic (PLEG): container finished" podID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerID="8c82898fbeee21507fcef8fff68b17d10099b7bdb5772551e282034624391627" exitCode=0
Oct 30 23:04:00.473259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.472777    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" event={"ID":"f91815b9-ea0e-4d30-b6fb-72e9933362cf","Type":"ContainerDied","Data":"8c82898fbeee21507fcef8fff68b17d10099b7bdb5772551e282034624391627"}
Oct 30 23:04:00.473259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.472842    4581 scope.go:122] "RemoveContainer" containerID="4bf414d2c5cc881ac7a528e9e4fa85158c7bc366f69a53ebe6066e5c5a669d49"
Oct 30 23:04:00.474771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.474749    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"
Oct 30 23:04:00.474879 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.474774    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx" event={"ID":"087d76fb-5e85-4885-9bca-2c0f91e8dff7","Type":"ContainerDied","Data":"53f08a93bc964f67b29e8572c5a53bfac956f57188d7e9c08312a10b2add845c"}
Oct 30 23:04:00.474879 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.474838    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53f08a93bc964f67b29e8572c5a53bfac956f57188d7e9c08312a10b2add845c"
Oct 30 23:04:00.602493 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.602459    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 23:04:00.644353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644324    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-socket\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644504    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-podinfo\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644546    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-kube-api-access-kkpsw\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-kube-api-access-kkpsw\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644572    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-envoy\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644614    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-socket" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644672    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-certs\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644720    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-credential-socket\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644768    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-data\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644811    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istiod-ca-cert\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.644897    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-token\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-token\") pod \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\" (UID: \"f91815b9-ea0e-4d30-b6fb-72e9933362cf\") "
Oct 30 23:04:00.645146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.645094    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-certs" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:00.645868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.645404    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.645868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.645426    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.645868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.645647    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-credential-socket" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:00.646040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.646022    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-data" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:00.646605 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.646579    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istiod-ca-cert" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:00.647903 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.647874    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-podinfo" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:00.648499 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.648455    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-envoy" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:00.649285 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.649239    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-kube-api-access-kkpsw" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "kube-api-access-kkpsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:00.649399 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.649377    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-token" pod "f91815b9-ea0e-4d30-b6fb-72e9933362cf" (UID: "f91815b9-ea0e-4d30-b6fb-72e9933362cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:00.682773 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.682735    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:04:00.746253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746204    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea21218-4240-428e-879a-14a7d982a62c-kube-api-access-7wr6d\" (UniqueName: \"kubernetes.io/projected/4ea21218-4240-428e-879a-14a7d982a62c-kube-api-access-7wr6d\") pod \"4ea21218-4240-428e-879a-14a7d982a62c\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") "
Oct 30 23:04:00.746531 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746273    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-openshift-ca\") pod \"4ea21218-4240-428e-879a-14a7d982a62c\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") "
Oct 30 23:04:00.746531 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746307    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-xks-ca\") pod \"4ea21218-4240-428e-879a-14a7d982a62c\" (UID: \"4ea21218-4240-428e-879a-14a7d982a62c\") "
Oct 30 23:04:00.746774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746720    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-openshift-ca" pod "4ea21218-4240-428e-879a-14a7d982a62c" (UID: "4ea21218-4240-428e-879a-14a7d982a62c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:00.746847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746811    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.746847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746830    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.746847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746839    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.746847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746847    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.747065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746854    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.747065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746862    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkpsw\" (UniqueName: \"kubernetes.io/projected/f91815b9-ea0e-4d30-b6fb-72e9933362cf-kube-api-access-kkpsw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.747065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.746871    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91815b9-ea0e-4d30-b6fb-72e9933362cf-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.749013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.748981    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea21218-4240-428e-879a-14a7d982a62c-kube-api-access-7wr6d" pod "4ea21218-4240-428e-879a-14a7d982a62c" (UID: "4ea21218-4240-428e-879a-14a7d982a62c"). InnerVolumeSpecName "kube-api-access-7wr6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:00.749711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.749683    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-xks-ca" pod "4ea21218-4240-428e-879a-14a7d982a62c" (UID: "4ea21218-4240-428e-879a-14a7d982a62c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:00.847816 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.847782    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wr6d\" (UniqueName: \"kubernetes.io/projected/4ea21218-4240-428e-879a-14a7d982a62c-kube-api-access-7wr6d\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.847816 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.847816    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:00.847816 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:00.847828    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea21218-4240-428e-879a-14a7d982a62c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:01.487523 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:01.487489    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"
Oct 30 23:04:01.487983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:01.487518    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns" event={"ID":"4ea21218-4240-428e-879a-14a7d982a62c","Type":"ContainerDied","Data":"9b157cbf3294aa6bc7cf165094ec996ad55eb5e9c4b2545289e89e3b40fa0a2d"}
Oct 30 23:04:01.487983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:01.487576    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b157cbf3294aa6bc7cf165094ec996ad55eb5e9c4b2545289e89e3b40fa0a2d"
Oct 30 23:04:01.489602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:01.489574    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5" event={"ID":"f91815b9-ea0e-4d30-b6fb-72e9933362cf","Type":"ContainerDied","Data":"713797c74f88ca5b2fb12aa58727ce334de1e605a86a89e28af6d9b5cc13edc5"}
Oct 30 23:04:01.489694 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:01.489617    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"
Oct 30 23:04:01.489694 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:01.489636    4581 scope.go:122] "RemoveContainer" containerID="8c82898fbeee21507fcef8fff68b17d10099b7bdb5772551e282034624391627"
Oct 30 23:04:01.547091 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:01.547059    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"]
Oct 30 23:04:01.551167 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:01.551137    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bed544-openshift-default-8659947965-dmmj5"]
Oct 30 23:04:02.096175 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.096143    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" path="/var/lib/kubelet/pods/f91815b9-ea0e-4d30-b6fb-72e9933362cf/volumes"
Oct 30 23:04:02.773120 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.773087    4581 scope.go:122] "RemoveContainer" containerID="44dd473da08842e1a93f7009e203219706763e4d849b34c7970abffc4af8bc0a"
Oct 30 23:04:02.785086 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.785057    4581 scope.go:122] "RemoveContainer" containerID="e542de2d24e81aadfc4cb5f7cf46e1c7ed0d326776bb700a1c2b320619afd00c"
Oct 30 23:04:02.798643 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.798621    4581 scope.go:122] "RemoveContainer" containerID="56524657ee5a4ca2f10562b72842a9964fef3b78d12b7196c8da49d71d3b50f2"
Oct 30 23:04:02.900244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900203    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"]
Oct 30 23:04:02.900805 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900786    4581 cpu_manager.go:414] "containerMap: removing container" podUID="087d76fb-5e85-4885-9bca-2c0f91e8dff7" containerName="revoke-keys"
Oct 30 23:04:02.900891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900807    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087d76fb-5e85-4885-9bca-2c0f91e8dff7" containerName="revoke-keys"
Oct 30 23:04:02.900891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900829    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4ea21218-4240-428e-879a-14a7d982a62c" containerName="revoke-keys"
Oct 30 23:04:02.900891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900837    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea21218-4240-428e-879a-14a7d982a62c" containerName="revoke-keys"
Oct 30 23:04:02.900891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900846    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy"
Oct 30 23:04:02.900891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900854    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy"
Oct 30 23:04:02.901144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900923    4581 cpu_manager.go:414] "containerMap: removing container" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy"
Oct 30 23:04:02.901144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.900932    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy"
Oct 30 23:04:02.901144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.901082    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="087d76fb-5e85-4885-9bca-2c0f91e8dff7" containerName="revoke-keys"
Oct 30 23:04:02.901144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.901099    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea21218-4240-428e-879a-14a7d982a62c" containerName="revoke-keys"
Oct 30 23:04:02.901144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.901111    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy"
Oct 30 23:04:02.901144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.901126    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="998ef2e0-9da0-4c59-8d85-27fc27427cc3" containerName="istio-proxy"
Oct 30 23:04:02.906872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.906850    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:02.925675 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.925640    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"]
Oct 30 23:04:02.926509 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.926403    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:04:02.926654 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.926539    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy"
Oct 30 23:04:02.926654 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.926555    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy"
Oct 30 23:04:02.926779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.926759    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91815b9-ea0e-4d30-b6fb-72e9933362cf" containerName="istio-proxy"
Oct 30 23:04:02.949530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.947903    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:04:02.949530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.948334    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:04:02.949530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.948470    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:04:02.965629 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.960078    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"]
Oct 30 23:04:02.965629 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.960127    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"]
Oct 30 23:04:02.965629 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.960710    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:02.975783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.973909    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:02.975783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.974125    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qnc\" (UniqueName: \"kubernetes.io/projected/ab171e36-3a54-4c31-bd6a-5d179092b14a-kube-api-access-92qnc\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:02.975783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.974315    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:02.975783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.974371    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntlgd\" (UniqueName: \"kubernetes.io/projected/81dfe808-2ff0-4d65-b910-50d112f89026-kube-api-access-ntlgd\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:02.975783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.974410    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:02.975783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:02.974456    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:03.075732 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.075613    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntlgd\" (UniqueName: \"kubernetes.io/projected/81dfe808-2ff0-4d65-b910-50d112f89026-kube-api-access-ntlgd\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:03.075732 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.075675    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:03.075732 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.075711    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:03.076048 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.075789    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:03.076048 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.075854    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qnc\" (UniqueName: \"kubernetes.io/projected/ab171e36-3a54-4c31-bd6a-5d179092b14a-kube-api-access-92qnc\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:03.076048 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.075917    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:03.076609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.076574    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:03.076899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.076876    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:03.078512 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.078491    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:03.078802 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.078786    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:03.095241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.095195    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qnc\" (UniqueName: \"kubernetes.io/projected/ab171e36-3a54-4c31-bd6a-5d179092b14a-kube-api-access-92qnc\") pod \"maas-api-revoke-sub-7acc14d9-gvwcs\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:03.095609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.095585    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntlgd\" (UniqueName: \"kubernetes.io/projected/81dfe808-2ff0-4d65-b910-50d112f89026-kube-api-access-ntlgd\") pod \"maas-api-revoke-sub-60728c12-szf49\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:03.226585 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.226551    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:03.279876 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.279843    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:03.541947 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.541917    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"]
Oct 30 23:04:03.542077 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:03.542054    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dfe808_2ff0_4d65_b910_50d112f89026.slice/crio-ba24a7b0dfe7522fb6458594648ad93dbd2c7aa795db9be5e578572b43f4c27a WatchSource:0}: Error finding container ba24a7b0dfe7522fb6458594648ad93dbd2c7aa795db9be5e578572b43f4c27a: Status 404 returned error can't find the container with id ba24a7b0dfe7522fb6458594648ad93dbd2c7aa795db9be5e578572b43f4c27a
Oct 30 23:04:03.615050 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.615022    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"]
Oct 30 23:04:03.615159 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:03.615113    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab171e36_3a54_4c31_bd6a_5d179092b14a.slice/crio-0344bf96108a6ba8c0819354637992ad25004176d70d312a1241c0547c7b01d9 WatchSource:0}: Error finding container 0344bf96108a6ba8c0819354637992ad25004176d70d312a1241c0547c7b01d9: Status 404 returned error can't find the container with id 0344bf96108a6ba8c0819354637992ad25004176d70d312a1241c0547c7b01d9
Oct 30 23:04:03.785500 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:03.785469    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"]
Oct 30 23:04:04.521674 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:04.521643    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs" event={"ID":"ab171e36-3a54-4c31-bd6a-5d179092b14a","Type":"ContainerStarted","Data":"28a2606219818752083164374df1c3bc118e741b121571844d35ea3a71160305"}
Oct 30 23:04:04.521914 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:04.521700    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs" event={"ID":"ab171e36-3a54-4c31-bd6a-5d179092b14a","Type":"ContainerStarted","Data":"0344bf96108a6ba8c0819354637992ad25004176d70d312a1241c0547c7b01d9"}
Oct 30 23:04:04.523442 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:04.523407    4581 generic.go:356] "Generic (PLEG): container finished" podID="81dfe808-2ff0-4d65-b910-50d112f89026" containerID="439fb731ee190db9be3c39982dc0ce9ebd3bb6a6b315fed8243f2e05c6d804ab" exitCode=0
Oct 30 23:04:04.523582 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:04.523483    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49" event={"ID":"81dfe808-2ff0-4d65-b910-50d112f89026","Type":"ContainerDied","Data":"439fb731ee190db9be3c39982dc0ce9ebd3bb6a6b315fed8243f2e05c6d804ab"}
Oct 30 23:04:04.523582 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:04.523532    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49" event={"ID":"81dfe808-2ff0-4d65-b910-50d112f89026","Type":"ContainerStarted","Data":"ba24a7b0dfe7522fb6458594648ad93dbd2c7aa795db9be5e578572b43f4c27a"}
Oct 30 23:04:04.569248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:04.569156    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs" podStartSLOduration=2.569134637 podStartE2EDuration="2.569134637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:04.563160492 +0000 UTC m=+3428.602099380" watchObservedRunningTime="2026-10-30 23:04:04.569134637 +0000 UTC m=+3428.608073526"
Oct 30 23:04:05.533032 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.532994    4581 generic.go:356] "Generic (PLEG): container finished" podID="ab171e36-3a54-4c31-bd6a-5d179092b14a" containerID="28a2606219818752083164374df1c3bc118e741b121571844d35ea3a71160305" exitCode=0
Oct 30 23:04:05.533515 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.533086    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs" event={"ID":"ab171e36-3a54-4c31-bd6a-5d179092b14a","Type":"ContainerDied","Data":"28a2606219818752083164374df1c3bc118e741b121571844d35ea3a71160305"}
Oct 30 23:04:05.730330 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.730304    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:05.810354 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.810245    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-xks-ca\") pod \"81dfe808-2ff0-4d65-b910-50d112f89026\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") "
Oct 30 23:04:05.810354 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.810339    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81dfe808-2ff0-4d65-b910-50d112f89026-kube-api-access-ntlgd\" (UniqueName: \"kubernetes.io/projected/81dfe808-2ff0-4d65-b910-50d112f89026-kube-api-access-ntlgd\") pod \"81dfe808-2ff0-4d65-b910-50d112f89026\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") "
Oct 30 23:04:05.810620 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.810477    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-openshift-ca\") pod \"81dfe808-2ff0-4d65-b910-50d112f89026\" (UID: \"81dfe808-2ff0-4d65-b910-50d112f89026\") "
Oct 30 23:04:05.810831 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.810803    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-openshift-ca" pod "81dfe808-2ff0-4d65-b910-50d112f89026" (UID: "81dfe808-2ff0-4d65-b910-50d112f89026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:05.811011 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.810987    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:05.813234 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.813187    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-xks-ca" pod "81dfe808-2ff0-4d65-b910-50d112f89026" (UID: "81dfe808-2ff0-4d65-b910-50d112f89026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:05.813336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.813208    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81dfe808-2ff0-4d65-b910-50d112f89026-kube-api-access-ntlgd" pod "81dfe808-2ff0-4d65-b910-50d112f89026" (UID: "81dfe808-2ff0-4d65-b910-50d112f89026"). InnerVolumeSpecName "kube-api-access-ntlgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:05.911933 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.911898    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dfe808-2ff0-4d65-b910-50d112f89026-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:05.911933 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:05.911939    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntlgd\" (UniqueName: \"kubernetes.io/projected/81dfe808-2ff0-4d65-b910-50d112f89026-kube-api-access-ntlgd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:06.304454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.304420    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"]
Oct 30 23:04:06.332016 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.331982    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"]
Oct 30 23:04:06.543724 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.543615    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc" podUID="9be81c22-942d-4049-b971-e55f040bdd92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:06.544287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.543717    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv" podUID="60eef6f3-d9eb-4716-bbb3-15f4e28249b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:06.548160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.548134    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"
Oct 30 23:04:06.548313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.548168    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49" event={"ID":"81dfe808-2ff0-4d65-b910-50d112f89026","Type":"ContainerDied","Data":"ba24a7b0dfe7522fb6458594648ad93dbd2c7aa795db9be5e578572b43f4c27a"}
Oct 30 23:04:06.548313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.548258    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba24a7b0dfe7522fb6458594648ad93dbd2c7aa795db9be5e578572b43f4c27a"
Oct 30 23:04:06.720928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.720905    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:06.824482 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.824408    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab171e36-3a54-4c31-bd6a-5d179092b14a-kube-api-access-92qnc\" (UniqueName: \"kubernetes.io/projected/ab171e36-3a54-4c31-bd6a-5d179092b14a-kube-api-access-92qnc\") pod \"ab171e36-3a54-4c31-bd6a-5d179092b14a\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") "
Oct 30 23:04:06.824639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.824492    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-xks-ca\") pod \"ab171e36-3a54-4c31-bd6a-5d179092b14a\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") "
Oct 30 23:04:06.824639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.824528    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-openshift-ca\") pod \"ab171e36-3a54-4c31-bd6a-5d179092b14a\" (UID: \"ab171e36-3a54-4c31-bd6a-5d179092b14a\") "
Oct 30 23:04:06.824930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.824901    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-openshift-ca" pod "ab171e36-3a54-4c31-bd6a-5d179092b14a" (UID: "ab171e36-3a54-4c31-bd6a-5d179092b14a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:06.826607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.826581    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-xks-ca" pod "ab171e36-3a54-4c31-bd6a-5d179092b14a" (UID: "ab171e36-3a54-4c31-bd6a-5d179092b14a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:06.826737 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.826646    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab171e36-3a54-4c31-bd6a-5d179092b14a-kube-api-access-92qnc" pod "ab171e36-3a54-4c31-bd6a-5d179092b14a" (UID: "ab171e36-3a54-4c31-bd6a-5d179092b14a"). InnerVolumeSpecName "kube-api-access-92qnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:06.944552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.944517    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92qnc\" (UniqueName: \"kubernetes.io/projected/ab171e36-3a54-4c31-bd6a-5d179092b14a-kube-api-access-92qnc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:06.944552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.944553    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:06.944786 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:06.944570    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab171e36-3a54-4c31-bd6a-5d179092b14a-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:07.100566 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.100536    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 23:04:07.133074 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.133049    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 23:04:07.148402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.148374    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be81c22-942d-4049-b971-e55f040bdd92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be81c22-942d-4049-b971-e55f040bdd92-plugins-config-volume\") pod \"9be81c22-942d-4049-b971-e55f040bdd92\" (UID: \"9be81c22-942d-4049-b971-e55f040bdd92\") "
Oct 30 23:04:07.148562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.148544    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be81c22-942d-4049-b971-e55f040bdd92-kube-api-access-xngtn\" (UniqueName: \"kubernetes.io/projected/9be81c22-942d-4049-b971-e55f040bdd92-kube-api-access-xngtn\") pod \"9be81c22-942d-4049-b971-e55f040bdd92\" (UID: \"9be81c22-942d-4049-b971-e55f040bdd92\") "
Oct 30 23:04:07.148718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.148694    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be81c22-942d-4049-b971-e55f040bdd92-plugins-config-volume" pod "9be81c22-942d-4049-b971-e55f040bdd92" (UID: "9be81c22-942d-4049-b971-e55f040bdd92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:07.149629 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.149599    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be81c22-942d-4049-b971-e55f040bdd92-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:07.151064 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.151025    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be81c22-942d-4049-b971-e55f040bdd92-kube-api-access-xngtn" pod "9be81c22-942d-4049-b971-e55f040bdd92" (UID: "9be81c22-942d-4049-b971-e55f040bdd92"). InnerVolumeSpecName "kube-api-access-xngtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:07.250949 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.250916    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-plugins-config-volume\") pod \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\" (UID: \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\") "
Oct 30 23:04:07.251128 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.251028    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-kube-api-access-jw9fl\" (UniqueName: \"kubernetes.io/projected/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-kube-api-access-jw9fl\") pod \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\" (UID: \"60eef6f3-d9eb-4716-bbb3-15f4e28249b8\") "
Oct 30 23:04:07.251416 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.251391    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-plugins-config-volume" pod "60eef6f3-d9eb-4716-bbb3-15f4e28249b8" (UID: "60eef6f3-d9eb-4716-bbb3-15f4e28249b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:07.251551 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.251534    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:07.251602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.251553    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngtn\" (UniqueName: \"kubernetes.io/projected/9be81c22-942d-4049-b971-e55f040bdd92-kube-api-access-xngtn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:07.253572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.253541    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-kube-api-access-jw9fl" pod "60eef6f3-d9eb-4716-bbb3-15f4e28249b8" (UID: "60eef6f3-d9eb-4716-bbb3-15f4e28249b8"). InnerVolumeSpecName "kube-api-access-jw9fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:07.357434 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.353404    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw9fl\" (UniqueName: \"kubernetes.io/projected/60eef6f3-d9eb-4716-bbb3-15f4e28249b8-kube-api-access-jw9fl\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:07.607383 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.606654    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs" event={"ID":"ab171e36-3a54-4c31-bd6a-5d179092b14a","Type":"ContainerDied","Data":"0344bf96108a6ba8c0819354637992ad25004176d70d312a1241c0547c7b01d9"}
Oct 30 23:04:07.607383 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.606723    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0344bf96108a6ba8c0819354637992ad25004176d70d312a1241c0547c7b01d9"
Oct 30 23:04:07.607383 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.606813    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"
Oct 30 23:04:07.611848 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.611766    4581 generic.go:356] "Generic (PLEG): container finished" podID="9be81c22-942d-4049-b971-e55f040bdd92" containerID="9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09" exitCode=0
Oct 30 23:04:07.612100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.611823    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"
Oct 30 23:04:07.612252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.611842    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc" event={"ID":"9be81c22-942d-4049-b971-e55f040bdd92","Type":"ContainerDied","Data":"9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09"}
Oct 30 23:04:07.612327 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.612306    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc" event={"ID":"9be81c22-942d-4049-b971-e55f040bdd92","Type":"ContainerDied","Data":"20002a512bf2591bf67791e353d601b1fdc00add37b5317ad886872dc951bb7f"}
Oct 30 23:04:07.612378 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.612360    4581 scope.go:122] "RemoveContainer" containerID="9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09"
Oct 30 23:04:07.627300 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.627263    4581 generic.go:356] "Generic (PLEG): container finished" podID="60eef6f3-d9eb-4716-bbb3-15f4e28249b8" containerID="0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0" exitCode=0
Oct 30 23:04:07.627451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.627346    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"
Oct 30 23:04:07.627451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.627357    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv" event={"ID":"60eef6f3-d9eb-4716-bbb3-15f4e28249b8","Type":"ContainerDied","Data":"0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0"}
Oct 30 23:04:07.627451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.627420    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv" event={"ID":"60eef6f3-d9eb-4716-bbb3-15f4e28249b8","Type":"ContainerDied","Data":"4c89aa71500a65543c1b0b824a05b88e05b79c4455a129f9aa85130781ca6078"}
Oct 30 23:04:07.649928 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.649903    4581 scope.go:122] "RemoveContainer" containerID="9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09"
Oct 30 23:04:07.650323 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:07.650287    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09\": container with ID starting with 9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09 not found: ID does not exist" containerID="9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09"
Oct 30 23:04:07.650463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.650330    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09"} err="failed to get container status \"9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09\": rpc error: code = NotFound desc = could not find container \"9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09\": container with ID starting with 9267aa9f4a4800fe65b6876b04b6739b5f6aeff02711623af0588074ced04e09 not found: ID does not exist"
Oct 30 23:04:07.650463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.650356    4581 scope.go:122] "RemoveContainer" containerID="0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0"
Oct 30 23:04:07.661647 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.661626    4581 scope.go:122] "RemoveContainer" containerID="0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0"
Oct 30 23:04:07.661990 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:07.661968    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0\": container with ID starting with 0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0 not found: ID does not exist" containerID="0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0"
Oct 30 23:04:07.662054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.662004    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0"} err="failed to get container status \"0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0\": rpc error: code = NotFound desc = could not find container \"0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0\": container with ID starting with 0bf79c93b363f7b6cd9b5294489e69ee1de25567d3e9f16e8587cd571c167ae0 not found: ID does not exist"
Oct 30 23:04:07.681043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.681013    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"]
Oct 30 23:04:07.687041 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.687011    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-54114e-5cf4644f-8htwc"]
Oct 30 23:04:07.728573 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.728542    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"]
Oct 30 23:04:07.732554 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.732527    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-54114e-7f6f89654c-sb4mv"]
Oct 30 23:04:07.775153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.775117    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"]
Oct 30 23:04:07.779192 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:07.779161    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3cd77f-rpbbk"]
Oct 30 23:04:08.100904 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:08.100822    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60eef6f3-d9eb-4716-bbb3-15f4e28249b8" path="/var/lib/kubelet/pods/60eef6f3-d9eb-4716-bbb3-15f4e28249b8/volumes"
Oct 30 23:04:08.101542 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:08.101520    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff10ba1-afd1-42d2-a842-264e22ae819e" path="/var/lib/kubelet/pods/6ff10ba1-afd1-42d2-a842-264e22ae819e/volumes"
Oct 30 23:04:08.102081 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:08.102062    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be81c22-942d-4049-b971-e55f040bdd92" path="/var/lib/kubelet/pods/9be81c22-942d-4049-b971-e55f040bdd92/volumes"
Oct 30 23:04:08.785976 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:08.785939    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw" podUID="cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a" containerName="payload-processing" containerID="cri-o://24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b" gracePeriod=25
Oct 30 23:04:09.143289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.143262    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 23:04:09.202840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.199287    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-kube-api-access-5zcch\" (UniqueName: \"kubernetes.io/projected/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-kube-api-access-5zcch\") pod \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\" (UID: \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\") "
Oct 30 23:04:09.202840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.199340    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-plugins-config-volume\") pod \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\" (UID: \"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a\") "
Oct 30 23:04:09.202840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.201133    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-plugins-config-volume" pod "cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a" (UID: "cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:09.203203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.202890    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-kube-api-access-5zcch" pod "cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a" (UID: "cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a"). InnerVolumeSpecName "kube-api-access-5zcch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:09.301174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.301136    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zcch\" (UniqueName: \"kubernetes.io/projected/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-kube-api-access-5zcch\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:09.301174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.301179    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:09.662967 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.662916    4581 generic.go:356] "Generic (PLEG): container finished" podID="cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a" containerID="24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b" exitCode=0
Oct 30 23:04:09.663309 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.663279    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw" event={"ID":"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a","Type":"ContainerDied","Data":"24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b"}
Oct 30 23:04:09.663469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.663374    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw" event={"ID":"cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a","Type":"ContainerDied","Data":"57fff09bc2dfafc73259676787673b4489d13bdd619153f0055082eb18f248a9"}
Oct 30 23:04:09.663469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.663417    4581 scope.go:122] "RemoveContainer" containerID="24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b"
Oct 30 23:04:09.663892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.663870    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"
Oct 30 23:04:09.675179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.675161    4581 scope.go:122] "RemoveContainer" containerID="24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b"
Oct 30 23:04:09.675501 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:09.675481    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b\": container with ID starting with 24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b not found: ID does not exist" containerID="24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b"
Oct 30 23:04:09.675556 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.675510    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b"} err="failed to get container status \"24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b\": rpc error: code = NotFound desc = could not find container \"24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b\": container with ID starting with 24f76bd62e5e4393749d122ff479395904341c3d75ba9242581ba3261cd5ee3b not found: ID does not exist"
Oct 30 23:04:09.718734 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.718702    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"]
Oct 30 23:04:09.727857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:09.727824    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6af86e-8bfcb675f-rgngw"]
Oct 30 23:04:10.099627 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:10.099539    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a" path="/var/lib/kubelet/pods/cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a/volumes"
Oct 30 23:04:10.817968 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:10.817928    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:04:11.304765 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:11.304716    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29" podUID="407ec7cd-8278-4a86-b8ff-0fb8500b531a" containerName="payload-processing" containerID="cri-o://1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097" gracePeriod=25
Oct 30 23:04:11.333027 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:11.332986    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r" podUID="9cd4230d-82fc-4310-a897-744c6deb6404" containerName="payload-pre-processing" containerID="cri-o://a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0" gracePeriod=25
Oct 30 23:04:11.839759 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:11.839719    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"]
Oct 30 23:04:11.839997 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:11.839970    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5" podUID="0e4c056b-ec3b-4201-8115-045031025b1b" containerName="payload-pre-processing" containerID="cri-o://c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d" gracePeriod=30
Oct 30 23:04:11.973816 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:11.973786    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"]
Oct 30 23:04:11.978166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:11.978136    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-012ba0-th4gx"]
Oct 30 23:04:12.099237 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.099135    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21135552-aa62-4db2-a43b-f9480fbbc6bd" path="/var/lib/kubelet/pods/21135552-aa62-4db2-a43b-f9480fbbc6bd/volumes"
Oct 30 23:04:12.132984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.132957    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:04:12.236439 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.236400    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4c056b-ec3b-4201-8115-045031025b1b-kube-api-access-d4vmm\" (UniqueName: \"kubernetes.io/projected/0e4c056b-ec3b-4201-8115-045031025b1b-kube-api-access-d4vmm\") pod \"0e4c056b-ec3b-4201-8115-045031025b1b\" (UID: \"0e4c056b-ec3b-4201-8115-045031025b1b\") "
Oct 30 23:04:12.237248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.237200    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4c056b-ec3b-4201-8115-045031025b1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4c056b-ec3b-4201-8115-045031025b1b-plugins-config-volume\") pod \"0e4c056b-ec3b-4201-8115-045031025b1b\" (UID: \"0e4c056b-ec3b-4201-8115-045031025b1b\") "
Oct 30 23:04:12.237657 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.237537    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4c056b-ec3b-4201-8115-045031025b1b-plugins-config-volume" pod "0e4c056b-ec3b-4201-8115-045031025b1b" (UID: "0e4c056b-ec3b-4201-8115-045031025b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:12.238233 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.238180    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4c056b-ec3b-4201-8115-045031025b1b-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.244784 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.240849    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4c056b-ec3b-4201-8115-045031025b1b-kube-api-access-d4vmm" pod "0e4c056b-ec3b-4201-8115-045031025b1b" (UID: "0e4c056b-ec3b-4201-8115-045031025b1b"). InnerVolumeSpecName "kube-api-access-d4vmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:12.339625 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.339586    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vmm\" (UniqueName: \"kubernetes.io/projected/0e4c056b-ec3b-4201-8115-045031025b1b-kube-api-access-d4vmm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.600975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.600933    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"]
Oct 30 23:04:12.604613 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.604590    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f94561-8dnns"]
Oct 30 23:04:12.704034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.703995    4581 generic.go:356] "Generic (PLEG): container finished" podID="0e4c056b-ec3b-4201-8115-045031025b1b" containerID="c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d" exitCode=0
Oct 30 23:04:12.704273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.704044    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"
Oct 30 23:04:12.704273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.704087    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5" event={"ID":"0e4c056b-ec3b-4201-8115-045031025b1b","Type":"ContainerDied","Data":"c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d"}
Oct 30 23:04:12.704273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.704151    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5" event={"ID":"0e4c056b-ec3b-4201-8115-045031025b1b","Type":"ContainerDied","Data":"534bfb9d089c8ca41266ba040c12aeff1dc46c55f24ffa5009dcd577996382ee"}
Oct 30 23:04:12.704273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.704182    4581 scope.go:122] "RemoveContainer" containerID="c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d"
Oct 30 23:04:12.715976 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.715958    4581 scope.go:122] "RemoveContainer" containerID="c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d"
Oct 30 23:04:12.716268 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:12.716249    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d\": container with ID starting with c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d not found: ID does not exist" containerID="c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d"
Oct 30 23:04:12.716348 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.716274    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d"} err="failed to get container status \"c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d\": rpc error: code = NotFound desc = could not find container \"c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d\": container with ID starting with c416b8d79d6451e6e1f964262ebb717c3e335ec120aa8b3a045ee8d8754f982d not found: ID does not exist"
Oct 30 23:04:12.750261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.750197    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"]
Oct 30 23:04:12.761332 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:12.761281    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f94561-66b74889b7-j6lk5"]
Oct 30 23:04:14.097982 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:14.097939    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4c056b-ec3b-4201-8115-045031025b1b" path="/var/lib/kubelet/pods/0e4c056b-ec3b-4201-8115-045031025b1b/volumes"
Oct 30 23:04:14.098615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:14.098595    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea21218-4240-428e-879a-14a7d982a62c" path="/var/lib/kubelet/pods/4ea21218-4240-428e-879a-14a7d982a62c/volumes"
Oct 30 23:04:14.781667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:14.781621    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29" podUID="407ec7cd-8278-4a86-b8ff-0fb8500b531a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:14.781879 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:14.781801    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r" podUID="9cd4230d-82fc-4310-a897-744c6deb6404" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:16.061826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.061788    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"]
Oct 30 23:04:16.062385 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.062031    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" containerID="cri-o://0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54" gracePeriod=30
Oct 30 23:04:16.622172 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.622130    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"]
Oct 30 23:04:16.626420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.626394    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-23aab2-6fntx"]
Oct 30 23:04:16.893913 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.893820    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"]
Oct 30 23:04:16.894369 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894355    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ab171e36-3a54-4c31-bd6a-5d179092b14a" containerName="revoke-keys"
Oct 30 23:04:16.894453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894370    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab171e36-3a54-4c31-bd6a-5d179092b14a" containerName="revoke-keys"
Oct 30 23:04:16.894453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894401    4581 cpu_manager.go:414] "containerMap: removing container" podUID="81dfe808-2ff0-4d65-b910-50d112f89026" containerName="revoke-keys"
Oct 30 23:04:16.894453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894407    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81dfe808-2ff0-4d65-b910-50d112f89026" containerName="revoke-keys"
Oct 30 23:04:16.894453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894427    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9be81c22-942d-4049-b971-e55f040bdd92" containerName="payload-processing"
Oct 30 23:04:16.894453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894432    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be81c22-942d-4049-b971-e55f040bdd92" containerName="payload-processing"
Oct 30 23:04:16.894453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894440    4581 cpu_manager.go:414] "containerMap: removing container" podUID="60eef6f3-d9eb-4716-bbb3-15f4e28249b8" containerName="payload-pre-processing"
Oct 30 23:04:16.894453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894445    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60eef6f3-d9eb-4716-bbb3-15f4e28249b8" containerName="payload-pre-processing"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894463    4581 cpu_manager.go:414] "containerMap: removing container" podUID="cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a" containerName="payload-processing"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894468    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a" containerName="payload-processing"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894477    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0e4c056b-ec3b-4201-8115-045031025b1b" containerName="payload-pre-processing"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894484    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4c056b-ec3b-4201-8115-045031025b1b" containerName="payload-pre-processing"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894566    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4c056b-ec3b-4201-8115-045031025b1b" containerName="payload-pre-processing"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894577    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab171e36-3a54-4c31-bd6a-5d179092b14a" containerName="revoke-keys"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894585    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="81dfe808-2ff0-4d65-b910-50d112f89026" containerName="revoke-keys"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894593    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be81c22-942d-4049-b971-e55f040bdd92" containerName="payload-processing"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894599    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="60eef6f3-d9eb-4716-bbb3-15f4e28249b8" containerName="payload-pre-processing"
Oct 30 23:04:16.894812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.894607    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8c98e1-9d5b-48f1-af1f-5b1ed4afab4a" containerName="payload-processing"
Oct 30 23:04:16.899621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.899599    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:16.918367 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.918340    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"]
Oct 30 23:04:16.924024 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.924001    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:04:16.924024 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.924009    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:04:16.926059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.926030    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:04:16.928378 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.928350    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:04:16.995533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.995495    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtgr\" (UniqueName: \"kubernetes.io/projected/ff27a071-df0b-4285-a2be-bbaef08bebe6-kube-api-access-jvtgr\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:16.995786 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.995556    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:16.996037 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:16.996015    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:17.097936 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.097902    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:17.098473 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.097956    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:17.098473 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.098055    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtgr\" (UniqueName: \"kubernetes.io/projected/ff27a071-df0b-4285-a2be-bbaef08bebe6-kube-api-access-jvtgr\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:17.098758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.098731    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:17.100681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.100643    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:17.117026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.116995    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtgr\" (UniqueName: \"kubernetes.io/projected/ff27a071-df0b-4285-a2be-bbaef08bebe6-kube-api-access-jvtgr\") pod \"maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:17.214281 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.214195    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:17.332272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.329910    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"]
Oct 30 23:04:17.335054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.335024    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6af86e-nl6h4"]
Oct 30 23:04:17.566403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.566373    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"]
Oct 30 23:04:17.568137 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:17.568107    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff27a071_df0b_4285_a2be_bbaef08bebe6.slice/crio-1a3122ba6d8303a9a7ced2e3106d5c717ceb4c745b985f628eb443851fe86b1c WatchSource:0}: Error finding container 1a3122ba6d8303a9a7ced2e3106d5c717ceb4c745b985f628eb443851fe86b1c: Status 404 returned error can't find the container with id 1a3122ba6d8303a9a7ced2e3106d5c717ceb4c745b985f628eb443851fe86b1c
Oct 30 23:04:17.751468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:17.751425    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x" event={"ID":"ff27a071-df0b-4285-a2be-bbaef08bebe6","Type":"ContainerStarted","Data":"1a3122ba6d8303a9a7ced2e3106d5c717ceb4c745b985f628eb443851fe86b1c"}
Oct 30 23:04:18.097965 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:18.097929    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087d76fb-5e85-4885-9bca-2c0f91e8dff7" path="/var/lib/kubelet/pods/087d76fb-5e85-4885-9bca-2c0f91e8dff7/volumes"
Oct 30 23:04:18.098420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:18.098405    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3aca8e4-2f2d-4801-a982-b99184ee2e8c" path="/var/lib/kubelet/pods/b3aca8e4-2f2d-4801-a982-b99184ee2e8c/volumes"
Oct 30 23:04:18.764297 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:18.764257    4581 generic.go:356] "Generic (PLEG): container finished" podID="ff27a071-df0b-4285-a2be-bbaef08bebe6" containerID="0f41a172454832f90edc71532b5178a0d048d4e90ba1e8bdca98ba907ca92e99" exitCode=0
Oct 30 23:04:18.764508 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:18.764350    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x" event={"ID":"ff27a071-df0b-4285-a2be-bbaef08bebe6","Type":"ContainerDied","Data":"0f41a172454832f90edc71532b5178a0d048d4e90ba1e8bdca98ba907ca92e99"}
Oct 30 23:04:19.746513 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:19.746464    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:04:19.747009 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:19.746539    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:04:19.943187 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:19.943158    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:20.035795 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.035649    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff27a071-df0b-4285-a2be-bbaef08bebe6-kube-api-access-jvtgr\" (UniqueName: \"kubernetes.io/projected/ff27a071-df0b-4285-a2be-bbaef08bebe6-kube-api-access-jvtgr\") pod \"ff27a071-df0b-4285-a2be-bbaef08bebe6\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") "
Oct 30 23:04:20.035954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.035853    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-xks-ca\") pod \"ff27a071-df0b-4285-a2be-bbaef08bebe6\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") "
Oct 30 23:04:20.035954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.035918    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-openshift-ca\") pod \"ff27a071-df0b-4285-a2be-bbaef08bebe6\" (UID: \"ff27a071-df0b-4285-a2be-bbaef08bebe6\") "
Oct 30 23:04:20.036635 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.036575    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-openshift-ca" pod "ff27a071-df0b-4285-a2be-bbaef08bebe6" (UID: "ff27a071-df0b-4285-a2be-bbaef08bebe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:20.042837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.039392    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-xks-ca" pod "ff27a071-df0b-4285-a2be-bbaef08bebe6" (UID: "ff27a071-df0b-4285-a2be-bbaef08bebe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:20.044123 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.044095    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff27a071-df0b-4285-a2be-bbaef08bebe6-kube-api-access-jvtgr" pod "ff27a071-df0b-4285-a2be-bbaef08bebe6" (UID: "ff27a071-df0b-4285-a2be-bbaef08bebe6"). InnerVolumeSpecName "kube-api-access-jvtgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:20.069350 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.069312    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"]
Oct 30 23:04:20.069881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.069847    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" containerID="cri-o://8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3" gracePeriod=30
Oct 30 23:04:20.137451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.137420    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.137451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.137448    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff27a071-df0b-4285-a2be-bbaef08bebe6-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.137451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.137459    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvtgr\" (UniqueName: \"kubernetes.io/projected/ff27a071-df0b-4285-a2be-bbaef08bebe6-kube-api-access-jvtgr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:20.757718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.757679    4581 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:04:20.758115 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.757739    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:04:20.784957 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.784920    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"
Oct 30 23:04:20.785154 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.784951    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x" event={"ID":"ff27a071-df0b-4285-a2be-bbaef08bebe6","Type":"ContainerDied","Data":"1a3122ba6d8303a9a7ced2e3106d5c717ceb4c745b985f628eb443851fe86b1c"}
Oct 30 23:04:20.785154 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.785007    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a3122ba6d8303a9a7ced2e3106d5c717ceb4c745b985f628eb443851fe86b1c"
Oct 30 23:04:20.876826 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.876787    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"]
Oct 30 23:04:20.877372 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.877354    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ff27a071-df0b-4285-a2be-bbaef08bebe6" containerName="revoke-keys"
Oct 30 23:04:20.877372 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.877370    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff27a071-df0b-4285-a2be-bbaef08bebe6" containerName="revoke-keys"
Oct 30 23:04:20.877556 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.877478    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff27a071-df0b-4285-a2be-bbaef08bebe6" containerName="revoke-keys"
Oct 30 23:04:20.880941 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.880917    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:20.894490 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.894461    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:04:20.894643 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.894518    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:04:20.895019 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.894995    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:04:20.895019 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.895001    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:04:20.921987 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.921905    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"]
Oct 30 23:04:20.946343 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.946303    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:20.946552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.946409    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwcb\" (UniqueName: \"kubernetes.io/projected/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-kube-api-access-kqwcb\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:20.946552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:20.946479    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:21.048315 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.048198    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:21.048315 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.048268    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwcb\" (UniqueName: \"kubernetes.io/projected/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-kube-api-access-kqwcb\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:21.048315 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.048290    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:21.048951 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.048928    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:21.050805 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.050784    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:21.066106 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.066074    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwcb\" (UniqueName: \"kubernetes.io/projected/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-kube-api-access-kqwcb\") pod \"maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:21.197450 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.197418    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:21.341467 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.341437    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 23:04:21.456941 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.456902    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-podinfo\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.457114 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.456984    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043e7603-d47f-43c3-aee4-a62b9c1922d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043e7603-d47f-43c3-aee4-a62b9c1922d2-istiod-ca-cert\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.457114 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457046    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-socket\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.457263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457172    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-certs\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.457341 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457262    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-credential-socket\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.457341 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457306    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-kube-api-access-4t6pg\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-kube-api-access-4t6pg\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.457454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457357    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-socket" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.457454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457412    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-envoy\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.457568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457458    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-token\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-token\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.457568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457479    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-certs" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457496    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043e7603-d47f-43c3-aee4-a62b9c1922d2-istiod-ca-cert" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457578    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-credential-socket" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457523    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-data\") pod \"043e7603-d47f-43c3-aee4-a62b9c1922d2\" (UID: \"043e7603-d47f-43c3-aee4-a62b9c1922d2\") "
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.457794    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-data" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.458241    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043e7603-d47f-43c3-aee4-a62b9c1922d2-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.458266    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.458276    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.458284    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.458313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.458294    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.460059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.460031    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-kube-api-access-4t6pg" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "kube-api-access-4t6pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:21.460148 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.460027    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-podinfo" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:21.460315 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.460284    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-envoy" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:21.460367 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.460320    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-token" pod "043e7603-d47f-43c3-aee4-a62b9c1922d2" (UID: "043e7603-d47f-43c3-aee4-a62b9c1922d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:21.514852 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.514819    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"]
Oct 30 23:04:21.516552 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:21.516525    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b45d7fd_d2a8_48bd_a2ef_feeb3b9ddea3.slice/crio-b831e9fb73e96d49248ae218e2e721fe1058ce1d2f13094e7ca81be3197bdcfd WatchSource:0}: Error finding container b831e9fb73e96d49248ae218e2e721fe1058ce1d2f13094e7ca81be3197bdcfd: Status 404 returned error can't find the container with id b831e9fb73e96d49248ae218e2e721fe1058ce1d2f13094e7ca81be3197bdcfd
Oct 30 23:04:21.549854 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.549823    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"]
Oct 30 23:04:21.559365 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.559336    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.559365 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.559363    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.559567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.559373    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043e7603-d47f-43c3-aee4-a62b9c1922d2-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.559567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.559382    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6pg\" (UniqueName: \"kubernetes.io/projected/043e7603-d47f-43c3-aee4-a62b9c1922d2-kube-api-access-4t6pg\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:21.560592 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.560568    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"]
Oct 30 23:04:21.794996 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.794951    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc" event={"ID":"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3","Type":"ContainerStarted","Data":"b831e9fb73e96d49248ae218e2e721fe1058ce1d2f13094e7ca81be3197bdcfd"}
Oct 30 23:04:21.797105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.797078    4581 generic.go:356] "Generic (PLEG): container finished" podID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerID="0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54" exitCode=0
Oct 30 23:04:21.797272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.797138    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"
Oct 30 23:04:21.797272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.797170    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" event={"ID":"043e7603-d47f-43c3-aee4-a62b9c1922d2","Type":"ContainerDied","Data":"0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54"}
Oct 30 23:04:21.797272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.797262    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7" event={"ID":"043e7603-d47f-43c3-aee4-a62b9c1922d2","Type":"ContainerDied","Data":"416686301e1c63e7ec017639939be9afb7c86c43d53187f5de159e6d280d9ad5"}
Oct 30 23:04:21.797402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.797296    4581 scope.go:122] "RemoveContainer" containerID="0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54"
Oct 30 23:04:21.810255 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.810206    4581 scope.go:122] "RemoveContainer" containerID="70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5"
Oct 30 23:04:21.820660 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.820642    4581 scope.go:122] "RemoveContainer" containerID="0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54"
Oct 30 23:04:21.820958 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:21.820934    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54\": container with ID starting with 0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54 not found: ID does not exist" containerID="0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54"
Oct 30 23:04:21.821040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.820970    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54"} err="failed to get container status \"0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54\": rpc error: code = NotFound desc = could not find container \"0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54\": container with ID starting with 0cfe567aeaad863b1bdd98fe479d01132f502becf8cadaa52f1aa305f7ab2a54 not found: ID does not exist"
Oct 30 23:04:21.821040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.820993    4581 scope.go:122] "RemoveContainer" containerID="70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5"
Oct 30 23:04:21.821254 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:21.821233    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5\": container with ID starting with 70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5 not found: ID does not exist" containerID="70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5"
Oct 30 23:04:21.821312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.821262    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5"} err="failed to get container status \"70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5\": rpc error: code = NotFound desc = could not find container \"70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5\": container with ID starting with 70cac0056439cb163715c241323455bd185570dd8639c07aba787b7c2fa2f0a5 not found: ID does not exist"
Oct 30 23:04:21.845371 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.845343    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"]
Oct 30 23:04:21.853451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:21.853422    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-52wf7"]
Oct 30 23:04:22.097595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:22.097556    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" path="/var/lib/kubelet/pods/043e7603-d47f-43c3-aee4-a62b9c1922d2/volumes"
Oct 30 23:04:22.810778 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:22.810687    4581 generic.go:356] "Generic (PLEG): container finished" podID="1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3" containerID="b5e3d09c52d4cd4258c454abf98218ce866c3c491f711feb41d1a41dd650814c" exitCode=0
Oct 30 23:04:22.811286 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:22.810839    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc" event={"ID":"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3","Type":"ContainerDied","Data":"b5e3d09c52d4cd4258c454abf98218ce866c3c491f711feb41d1a41dd650814c"}
Oct 30 23:04:23.875199 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:23.874292    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"]
Oct 30 23:04:23.875199 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:23.874537    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" containerID="cri-o://857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6" gracePeriod=30
Oct 30 23:04:24.036506 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.036483    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:24.107087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.107060    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-openshift-ca\") pod \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") "
Oct 30 23:04:24.107299 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.107099    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-xks-ca\") pod \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") "
Oct 30 23:04:24.107299 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.107149    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-kube-api-access-kqwcb\" (UniqueName: \"kubernetes.io/projected/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-kube-api-access-kqwcb\") pod \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\" (UID: \"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3\") "
Oct 30 23:04:24.107520 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.107495    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-openshift-ca" pod "1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3" (UID: "1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:24.107622 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.107607    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:24.110372 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.110339    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-xks-ca" pod "1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3" (UID: "1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:24.110372 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.110352    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-kube-api-access-kqwcb" pod "1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3" (UID: "1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3"). InnerVolumeSpecName "kube-api-access-kqwcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:24.209183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.209154    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:24.209183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.209183    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwcb\" (UniqueName: \"kubernetes.io/projected/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3-kube-api-access-kqwcb\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:24.620860 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.620595    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"]
Oct 30 23:04:24.621182 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621166    4581 cpu_manager.go:414] "containerMap: removing container" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy"
Oct 30 23:04:24.621271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621184    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy"
Oct 30 23:04:24.621271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621195    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3" containerName="revoke-keys"
Oct 30 23:04:24.621271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621201    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3" containerName="revoke-keys"
Oct 30 23:04:24.621271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621236    4581 cpu_manager.go:414] "containerMap: removing container" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy"
Oct 30 23:04:24.621271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621241    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy"
Oct 30 23:04:24.621428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621338    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy"
Oct 30 23:04:24.621428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621347    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3" containerName="revoke-keys"
Oct 30 23:04:24.621428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.621356    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="043e7603-d47f-43c3-aee4-a62b9c1922d2" containerName="istio-proxy"
Oct 30 23:04:24.624772 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.624751    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.645065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.645035    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"]
Oct 30 23:04:24.714581 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.714546    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.714789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.714612    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kqz\" (UniqueName: \"kubernetes.io/projected/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-kube-api-access-k5kqz\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.714789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.714681    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.781701 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.781655    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29" podUID="407ec7cd-8278-4a86-b8ff-0fb8500b531a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:24.781892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.781172    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r" podUID="9cd4230d-82fc-4310-a897-744c6deb6404" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:24.816602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.816558    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.816840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.816652    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kqz\" (UniqueName: \"kubernetes.io/projected/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-kube-api-access-k5kqz\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.816840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.816686    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.817772 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.817724    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.819939 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.819910    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.832020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.831993    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"
Oct 30 23:04:24.832020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.832004    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc" event={"ID":"1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3","Type":"ContainerDied","Data":"b831e9fb73e96d49248ae218e2e721fe1058ce1d2f13094e7ca81be3197bdcfd"}
Oct 30 23:04:24.832297 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.832089    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b831e9fb73e96d49248ae218e2e721fe1058ce1d2f13094e7ca81be3197bdcfd"
Oct 30 23:04:24.835324 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.835300    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kqz\" (UniqueName: \"kubernetes.io/projected/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-kube-api-access-k5kqz\") pod \"maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:24.940066 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:24.939986    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:25.262032 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.261977    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"]
Oct 30 23:04:25.403203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.403144    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 23:04:25.524699 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.524664    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-podinfo\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.524884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.524802    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-token\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-token\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.524884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.524859    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-credential-socket\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.524979 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.524888    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-data\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.524979 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.524940    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istiod-ca-cert\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.525076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.524985    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-socket\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.525076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525014    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-envoy\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.525076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525052    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-certs\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.525273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525114    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-kube-api-access-8ps2f\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-kube-api-access-8ps2f\") pod \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\" (UID: \"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5\") "
Oct 30 23:04:25.525273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525207    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-credential-socket" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.525448 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525390    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-data" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.525448 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525398    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istiod-ca-cert" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:25.525708 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525687    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.525781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525715    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.525781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525730    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.525781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525733    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-socket" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.525781 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.525764    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-certs" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.551721 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.551677    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-podinfo" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:25.551721 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.551711    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-envoy" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.551980 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.551742    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-token" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:25.552349 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.552321    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-kube-api-access-8ps2f" pod "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" (UID: "caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5"). InnerVolumeSpecName "kube-api-access-8ps2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:25.629232 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.629168    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.632306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.632280    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.632477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.632319    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.632477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.632335    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.632477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.632349    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ps2f\" (UniqueName: \"kubernetes.io/projected/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-kube-api-access-8ps2f\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.632477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.632364    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:25.844332 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.844200    4581 generic.go:356] "Generic (PLEG): container finished" podID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerID="8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3" exitCode=0
Oct 30 23:04:25.844332 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.844272    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" event={"ID":"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5","Type":"ContainerDied","Data":"8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3"}
Oct 30 23:04:25.844332 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.844291    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"
Oct 30 23:04:25.844616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.844372    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls" event={"ID":"caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5","Type":"ContainerDied","Data":"072b9d4d426a9c4a511a015fbb7b65012f332d349db6c2d88bd9e26dce6f776a"}
Oct 30 23:04:25.844616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.844402    4581 scope.go:122] "RemoveContainer" containerID="8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3"
Oct 30 23:04:25.846648 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.846616    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw" event={"ID":"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd","Type":"ContainerStarted","Data":"2c9954c219d1ffc727fa4678c7a3e7351beda9ae7a16be604da094b06bbe25c7"}
Oct 30 23:04:25.846772 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.846692    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw" event={"ID":"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd","Type":"ContainerStarted","Data":"4ae2331593bdad4ecc49fce04e8e12253be7076d1dfa2445bc856f094ff753f2"}
Oct 30 23:04:25.856412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.856395    4581 scope.go:122] "RemoveContainer" containerID="171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a"
Oct 30 23:04:25.869631 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.869608    4581 scope.go:122] "RemoveContainer" containerID="8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3"
Oct 30 23:04:25.869995 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:25.869973    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3\": container with ID starting with 8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3 not found: ID does not exist" containerID="8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3"
Oct 30 23:04:25.870090 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.870007    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3"} err="failed to get container status \"8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3\": rpc error: code = NotFound desc = could not find container \"8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3\": container with ID starting with 8f0925e21c057740646a2092fc375ec827640a9abef2fd8a4067b7995e87a0d3 not found: ID does not exist"
Oct 30 23:04:25.870090 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.870026    4581 scope.go:122] "RemoveContainer" containerID="171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a"
Oct 30 23:04:25.870308 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:25.870288    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a\": container with ID starting with 171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a not found: ID does not exist" containerID="171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a"
Oct 30 23:04:25.870382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.870315    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a"} err="failed to get container status \"171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a\": rpc error: code = NotFound desc = could not find container \"171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a\": container with ID starting with 171a7ac07238fc754cb8911a175abb6f3ef0f6129f3ef10e55ce1395f8bac61a not found: ID does not exist"
Oct 30 23:04:25.893761 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.893730    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"]
Oct 30 23:04:25.906802 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.906772    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-kzdls"]
Oct 30 23:04:25.938196 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:25.938132    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw" podStartSLOduration=1.938113956 podStartE2EDuration="1.938113956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:25.933146156 +0000 UTC m=+3449.972085043" watchObservedRunningTime="2026-10-30 23:04:25.938113956 +0000 UTC m=+3449.977052842"
Oct 30 23:04:26.029145 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.029114    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"]
Oct 30 23:04:26.033181 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.033152    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fab2ecd-5zz9w"]
Oct 30 23:04:26.097711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.097635    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f2c635-a735-44fa-980e-9ec696ff8386" path="/var/lib/kubelet/pods/b7f2c635-a735-44fa-980e-9ec696ff8386/volumes"
Oct 30 23:04:26.098109 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.098096    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" path="/var/lib/kubelet/pods/caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5/volumes"
Oct 30 23:04:26.139342 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.139312    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"]
Oct 30 23:04:26.139837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.139823    4581 cpu_manager.go:414] "containerMap: removing container" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy"
Oct 30 23:04:26.139881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.139838    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy"
Oct 30 23:04:26.139881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.139848    4581 cpu_manager.go:414] "containerMap: removing container" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy"
Oct 30 23:04:26.139881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.139854    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy"
Oct 30 23:04:26.140008 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.139942    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy"
Oct 30 23:04:26.140008 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.139949    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="caaf4d3a-96f8-40c3-b0e8-03e6df1d6fc5" containerName="istio-proxy"
Oct 30 23:04:26.144932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.144909    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.164280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.164253    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"]
Oct 30 23:04:26.239785 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.239737    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwb4\" (UniqueName: \"kubernetes.io/projected/014e36ec-128f-4ae2-b880-99b1bef435eb-kube-api-access-mjwb4\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.239983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.239827    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.240054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.240015    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.341845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.341812    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.342017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.341899    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwb4\" (UniqueName: \"kubernetes.io/projected/014e36ec-128f-4ae2-b880-99b1bef435eb-kube-api-access-mjwb4\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.342017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.342010    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.343142 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.342592    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.344864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.344839    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.359064 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.359035    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwb4\" (UniqueName: \"kubernetes.io/projected/014e36ec-128f-4ae2-b880-99b1bef435eb-kube-api-access-mjwb4\") pod \"maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.460170 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.460133    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:26.781775 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.781744    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"]
Oct 30 23:04:26.784206 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:26.784174    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014e36ec_128f_4ae2_b880_99b1bef435eb.slice/crio-767f61969a247296d9a104a5aa5fbbb1b8bab0cce23cf55364f053817a8840c3 WatchSource:0}: Error finding container 767f61969a247296d9a104a5aa5fbbb1b8bab0cce23cf55364f053817a8840c3: Status 404 returned error can't find the container with id 767f61969a247296d9a104a5aa5fbbb1b8bab0cce23cf55364f053817a8840c3
Oct 30 23:04:26.857934 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.857895    4581 generic.go:356] "Generic (PLEG): container finished" podID="81ba0d7d-e3a7-4118-9b63-9cf132a47fbd" containerID="2c9954c219d1ffc727fa4678c7a3e7351beda9ae7a16be604da094b06bbe25c7" exitCode=0
Oct 30 23:04:26.858127 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.857935    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw" event={"ID":"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd","Type":"ContainerDied","Data":"2c9954c219d1ffc727fa4678c7a3e7351beda9ae7a16be604da094b06bbe25c7"}
Oct 30 23:04:26.859518 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:26.859490    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds" event={"ID":"014e36ec-128f-4ae2-b880-99b1bef435eb","Type":"ContainerStarted","Data":"767f61969a247296d9a104a5aa5fbbb1b8bab0cce23cf55364f053817a8840c3"}
Oct 30 23:04:27.400702 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:27.400661    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"]
Oct 30 23:04:27.430513 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:27.430434    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"]
Oct 30 23:04:27.874054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:27.874020    4581 generic.go:356] "Generic (PLEG): container finished" podID="014e36ec-128f-4ae2-b880-99b1bef435eb" containerID="e10eb43a95efaf639bd0f1d7637922eef09e7c137414e7d61bfb00593356daed" exitCode=0
Oct 30 23:04:27.874298 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:27.874158    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds" event={"ID":"014e36ec-128f-4ae2-b880-99b1bef435eb","Type":"ContainerDied","Data":"e10eb43a95efaf639bd0f1d7637922eef09e7c137414e7d61bfb00593356daed"}
Oct 30 23:04:28.034346 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.034321    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:28.088369 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.088334    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"]
Oct 30 23:04:28.119707 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.119676    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-54114e-hmt2w"]
Oct 30 23:04:28.195562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.195530    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-kube-api-access-k5kqz\" (UniqueName: \"kubernetes.io/projected/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-kube-api-access-k5kqz\") pod \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") "
Oct 30 23:04:28.195723 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.195653    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-xks-ca\") pod \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") "
Oct 30 23:04:28.195723 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.195681    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-openshift-ca\") pod \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\" (UID: \"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd\") "
Oct 30 23:04:28.196153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.196124    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-openshift-ca" pod "81ba0d7d-e3a7-4118-9b63-9cf132a47fbd" (UID: "81ba0d7d-e3a7-4118-9b63-9cf132a47fbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:28.198429 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.198398    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-kube-api-access-k5kqz" pod "81ba0d7d-e3a7-4118-9b63-9cf132a47fbd" (UID: "81ba0d7d-e3a7-4118-9b63-9cf132a47fbd"). InnerVolumeSpecName "kube-api-access-k5kqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:28.199075 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.199048    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-xks-ca" pod "81ba0d7d-e3a7-4118-9b63-9cf132a47fbd" (UID: "81ba0d7d-e3a7-4118-9b63-9cf132a47fbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:28.297292 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.297179    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:28.297292 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.297247    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:28.297292 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.297265    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5kqz\" (UniqueName: \"kubernetes.io/projected/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd-kube-api-access-k5kqz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:28.839857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.839815    4581 patch_prober.go:29] interesting pod/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:04:28.840276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.839879    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:04:28.888616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.888583    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"
Oct 30 23:04:28.888616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.888607    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw" event={"ID":"81ba0d7d-e3a7-4118-9b63-9cf132a47fbd","Type":"ContainerDied","Data":"4ae2331593bdad4ecc49fce04e8e12253be7076d1dfa2445bc856f094ff753f2"}
Oct 30 23:04:28.888845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:28.888666    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae2331593bdad4ecc49fce04e8e12253be7076d1dfa2445bc856f094ff753f2"
Oct 30 23:04:29.043347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.043318    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:29.167902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.167876    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 23:04:29.207766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.207730    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-xks-ca\") pod \"014e36ec-128f-4ae2-b880-99b1bef435eb\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") "
Oct 30 23:04:29.207982 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.207813    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-openshift-ca\") pod \"014e36ec-128f-4ae2-b880-99b1bef435eb\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") "
Oct 30 23:04:29.207982 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.207942    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014e36ec-128f-4ae2-b880-99b1bef435eb-kube-api-access-mjwb4\" (UniqueName: \"kubernetes.io/projected/014e36ec-128f-4ae2-b880-99b1bef435eb-kube-api-access-mjwb4\") pod \"014e36ec-128f-4ae2-b880-99b1bef435eb\" (UID: \"014e36ec-128f-4ae2-b880-99b1bef435eb\") "
Oct 30 23:04:29.209171 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.208713    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-openshift-ca" pod "014e36ec-128f-4ae2-b880-99b1bef435eb" (UID: "014e36ec-128f-4ae2-b880-99b1bef435eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:29.211033 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.211003    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-xks-ca" pod "014e36ec-128f-4ae2-b880-99b1bef435eb" (UID: "014e36ec-128f-4ae2-b880-99b1bef435eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:29.211152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.211031    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014e36ec-128f-4ae2-b880-99b1bef435eb-kube-api-access-mjwb4" pod "014e36ec-128f-4ae2-b880-99b1bef435eb" (UID: "014e36ec-128f-4ae2-b880-99b1bef435eb"). InnerVolumeSpecName "kube-api-access-mjwb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:29.309169 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309136    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-credential-socket\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309207    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-envoy\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309268    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-token\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-token\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309301    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2754d3d-bedb-443c-95a2-db9070f4ed02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2754d3d-bedb-443c-95a2-db9070f4ed02-istiod-ca-cert\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309376 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309360    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-podinfo\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309383    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-kube-api-access-svqk2\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-kube-api-access-svqk2\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309410    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-certs\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309519    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-data\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309539    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-credential-socket" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:29.309600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309559    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-socket\") pod \"e2754d3d-bedb-443c-95a2-db9070f4ed02\" (UID: \"e2754d3d-bedb-443c-95a2-db9070f4ed02\") "
Oct 30 23:04:29.309881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309787    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2754d3d-bedb-443c-95a2-db9070f4ed02-istiod-ca-cert" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:29.309881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309796    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-certs" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:29.309994 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309890    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-data" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:29.309994 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.309974    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-socket" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:29.310115 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.310085    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.310115 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.310102    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.310253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.310116    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014e36ec-128f-4ae2-b880-99b1bef435eb-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.310253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.310129    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.310253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.310139    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.310253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.310147    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwb4\" (UniqueName: \"kubernetes.io/projected/014e36ec-128f-4ae2-b880-99b1bef435eb-kube-api-access-mjwb4\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.310253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.310156    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.310253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.310164    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2754d3d-bedb-443c-95a2-db9070f4ed02-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.311569 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.311549    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-token" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:29.311890 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.311872    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-kube-api-access-svqk2" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "kube-api-access-svqk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:29.312126 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.312101    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-envoy" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:29.312126 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.312114    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-podinfo" pod "e2754d3d-bedb-443c-95a2-db9070f4ed02" (UID: "e2754d3d-bedb-443c-95a2-db9070f4ed02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:29.335792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.335768    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"]
Oct 30 23:04:29.411588 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.411503    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svqk2\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-kube-api-access-svqk2\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.411588 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.411534    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.411588 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.411545    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.411588 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.411553    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2754d3d-bedb-443c-95a2-db9070f4ed02-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:29.783620 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.783524    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"]
Oct 30 23:04:29.784330 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784310    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy"
Oct 30 23:04:29.784407 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784332    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy"
Oct 30 23:04:29.784407 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784394    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy"
Oct 30 23:04:29.784407 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784402    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy"
Oct 30 23:04:29.784572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784419    4581 cpu_manager.go:414] "containerMap: removing container" podUID="014e36ec-128f-4ae2-b880-99b1bef435eb" containerName="revoke-keys"
Oct 30 23:04:29.784572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784426    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014e36ec-128f-4ae2-b880-99b1bef435eb" containerName="revoke-keys"
Oct 30 23:04:29.784572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784442    4581 cpu_manager.go:414] "containerMap: removing container" podUID="81ba0d7d-e3a7-4118-9b63-9cf132a47fbd" containerName="revoke-keys"
Oct 30 23:04:29.784572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784449    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ba0d7d-e3a7-4118-9b63-9cf132a47fbd" containerName="revoke-keys"
Oct 30 23:04:29.784711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784575    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy"
Oct 30 23:04:29.784711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784588    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="014e36ec-128f-4ae2-b880-99b1bef435eb" containerName="revoke-keys"
Oct 30 23:04:29.784711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784599    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ba0d7d-e3a7-4118-9b63-9cf132a47fbd" containerName="revoke-keys"
Oct 30 23:04:29.784711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.784611    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerName="istio-proxy"
Oct 30 23:04:29.789050 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.789026    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:29.802150 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.802121    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"]
Oct 30 23:04:29.806966 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.806941    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:29.807117 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.807098    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"]
Oct 30 23:04:29.828155 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.828110    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"]
Oct 30 23:04:29.898372 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.898343    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"
Oct 30 23:04:29.898372 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.898361    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds" event={"ID":"014e36ec-128f-4ae2-b880-99b1bef435eb","Type":"ContainerDied","Data":"767f61969a247296d9a104a5aa5fbbb1b8bab0cce23cf55364f053817a8840c3"}
Oct 30 23:04:29.898845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.898418    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="767f61969a247296d9a104a5aa5fbbb1b8bab0cce23cf55364f053817a8840c3"
Oct 30 23:04:29.900649 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.900621    4581 generic.go:356] "Generic (PLEG): container finished" podID="e2754d3d-bedb-443c-95a2-db9070f4ed02" containerID="857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6" exitCode=0
Oct 30 23:04:29.900797 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.900669    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"
Oct 30 23:04:29.900797 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.900690    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" event={"ID":"e2754d3d-bedb-443c-95a2-db9070f4ed02","Type":"ContainerDied","Data":"857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6"}
Oct 30 23:04:29.900797 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.900738    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp" event={"ID":"e2754d3d-bedb-443c-95a2-db9070f4ed02","Type":"ContainerDied","Data":"b527b46911f1df4c2dc5af03f373296a9e248731aced5d74761531c1c61ccf50"}
Oct 30 23:04:29.900797 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.900770    4581 scope.go:122] "RemoveContainer" containerID="857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6"
Oct 30 23:04:29.912704 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.912690    4581 scope.go:122] "RemoveContainer" containerID="c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b"
Oct 30 23:04:29.917085 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.917067    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:29.917167 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.917121    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:29.917263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.917245    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwd7f\" (UniqueName: \"kubernetes.io/projected/88c35483-4333-4938-9d5f-ed2e7330b588-kube-api-access-nwd7f\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:29.917312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.917281    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:29.917398 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.917379    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:29.917468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.917454    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhqx\" (UniqueName: \"kubernetes.io/projected/cb6495fe-1800-42a5-9b56-04c4b5a74745-kube-api-access-8bhqx\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:29.923681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.923661    4581 scope.go:122] "RemoveContainer" containerID="857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6"
Oct 30 23:04:29.924004 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:29.923976    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6\": container with ID starting with 857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6 not found: ID does not exist" containerID="857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6"
Oct 30 23:04:29.924092 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.924011    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6"} err="failed to get container status \"857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6\": rpc error: code = NotFound desc = could not find container \"857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6\": container with ID starting with 857a1b50dbf366c84c702be16d09f08ab8c2870925f62afe2add8d38eef1a1d6 not found: ID does not exist"
Oct 30 23:04:29.924092 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.924030    4581 scope.go:122] "RemoveContainer" containerID="c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b"
Oct 30 23:04:29.924299 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:29.924277    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b\": container with ID starting with c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b not found: ID does not exist" containerID="c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b"
Oct 30 23:04:29.924354 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.924308    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b"} err="failed to get container status \"c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b\": rpc error: code = NotFound desc = could not find container \"c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b\": container with ID starting with c757ed18f9b49797dec9761f7820ea3621b1d6a9dbef853babe7fa5c15f0841b not found: ID does not exist"
Oct 30 23:04:29.960749 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.960717    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"]
Oct 30 23:04:29.966156 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:29.966129    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-23aab2-openshift-default-86c4fcc59c-qvpxp"]
Oct 30 23:04:30.018549 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.018520    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:30.018695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.018595    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:30.018695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.018656    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwd7f\" (UniqueName: \"kubernetes.io/projected/88c35483-4333-4938-9d5f-ed2e7330b588-kube-api-access-nwd7f\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:30.018695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.018684    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:30.018902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.018726    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:30.018902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.018809    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhqx\" (UniqueName: \"kubernetes.io/projected/cb6495fe-1800-42a5-9b56-04c4b5a74745-kube-api-access-8bhqx\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:30.019355 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.019333    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:30.019640 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.019369    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:30.021511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.021329    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:30.021511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.021380    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:30.038002 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.037945    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwd7f\" (UniqueName: \"kubernetes.io/projected/88c35483-4333-4938-9d5f-ed2e7330b588-kube-api-access-nwd7f\") pod \"maas-api-revoke-sub-5d21908a-w2fxm\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:30.038147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.038125    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhqx\" (UniqueName: \"kubernetes.io/projected/cb6495fe-1800-42a5-9b56-04c4b5a74745-kube-api-access-8bhqx\") pod \"maas-api-revoke-sub-704e8faf-qqgrn\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:30.097532 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.097490    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b31f3e8-f069-4538-8fd8-3550effd3903" path="/var/lib/kubelet/pods/2b31f3e8-f069-4538-8fd8-3550effd3903/volumes"
Oct 30 23:04:30.098157 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.098130    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2754d3d-bedb-443c-95a2-db9070f4ed02" path="/var/lib/kubelet/pods/e2754d3d-bedb-443c-95a2-db9070f4ed02/volumes"
Oct 30 23:04:30.105459 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.105435    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:30.123142 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.123049    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:30.423905 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.423852    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"]
Oct 30 23:04:30.439077 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:30.439048    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c35483_4333_4938_9d5f_ed2e7330b588.slice/crio-e08e9b506fe478473f84033c11d31d101ad34ea7bfb5ce5ba941bd24b1a77c9d WatchSource:0}: Error finding container e08e9b506fe478473f84033c11d31d101ad34ea7bfb5ce5ba941bd24b1a77c9d: Status 404 returned error can't find the container with id e08e9b506fe478473f84033c11d31d101ad34ea7bfb5ce5ba941bd24b1a77c9d
Oct 30 23:04:30.469427 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.469393    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"]
Oct 30 23:04:30.469869 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:30.469842    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6495fe_1800_42a5_9b56_04c4b5a74745.slice/crio-f914545c9ef7674f1d9dd7ba05417716506c4f073f5144cdc353549e139e83aa WatchSource:0}: Error finding container f914545c9ef7674f1d9dd7ba05417716506c4f073f5144cdc353549e139e83aa: Status 404 returned error can't find the container with id f914545c9ef7674f1d9dd7ba05417716506c4f073f5144cdc353549e139e83aa
Oct 30 23:04:30.890858 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.890825    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"]
Oct 30 23:04:30.921619 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.921573    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn" event={"ID":"cb6495fe-1800-42a5-9b56-04c4b5a74745","Type":"ContainerStarted","Data":"38dc3567ea9a92df8856e6763bcabf3d5955de0dacd6b0e0629cbee6b1a20e60"}
Oct 30 23:04:30.922029 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.921674    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn" event={"ID":"cb6495fe-1800-42a5-9b56-04c4b5a74745","Type":"ContainerStarted","Data":"f914545c9ef7674f1d9dd7ba05417716506c4f073f5144cdc353549e139e83aa"}
Oct 30 23:04:30.923481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.923452    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm" event={"ID":"88c35483-4333-4938-9d5f-ed2e7330b588","Type":"ContainerStarted","Data":"b603428e49d4020febacc5e18e2edf1ff8131dc64b8e0524cdf257ff1043ff7c"}
Oct 30 23:04:30.923603 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.923504    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm" event={"ID":"88c35483-4333-4938-9d5f-ed2e7330b588","Type":"ContainerStarted","Data":"e08e9b506fe478473f84033c11d31d101ad34ea7bfb5ce5ba941bd24b1a77c9d"}
Oct 30 23:04:30.976990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:30.976925    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm" podStartSLOduration=1.9769066180000001 podStartE2EDuration="1.976906618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:30.971327654 +0000 UTC m=+3455.010266540" watchObservedRunningTime="2026-10-30 23:04:30.976906618 +0000 UTC m=+3455.015845505"
Oct 30 23:04:31.010054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.010000    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn" podStartSLOduration=2.009985026 podStartE2EDuration="2.009985026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:31.00593125 +0000 UTC m=+3455.044870137" watchObservedRunningTime="2026-10-30 23:04:31.009985026 +0000 UTC m=+3455.048923932"
Oct 30 23:04:31.609631 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.609605    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 23:04:31.642882 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.642855    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 23:04:31.738522 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.738485    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407ec7cd-8278-4a86-b8ff-0fb8500b531a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407ec7cd-8278-4a86-b8ff-0fb8500b531a-plugins-config-volume\") pod \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\" (UID: \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\") "
Oct 30 23:04:31.738677 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.738614    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407ec7cd-8278-4a86-b8ff-0fb8500b531a-kube-api-access-6dqk5\" (UniqueName: \"kubernetes.io/projected/407ec7cd-8278-4a86-b8ff-0fb8500b531a-kube-api-access-6dqk5\") pod \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\" (UID: \"407ec7cd-8278-4a86-b8ff-0fb8500b531a\") "
Oct 30 23:04:31.738846 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.738824    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407ec7cd-8278-4a86-b8ff-0fb8500b531a-plugins-config-volume" pod "407ec7cd-8278-4a86-b8ff-0fb8500b531a" (UID: "407ec7cd-8278-4a86-b8ff-0fb8500b531a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:31.739003 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.738982    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407ec7cd-8278-4a86-b8ff-0fb8500b531a-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:31.740880 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.740857    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407ec7cd-8278-4a86-b8ff-0fb8500b531a-kube-api-access-6dqk5" pod "407ec7cd-8278-4a86-b8ff-0fb8500b531a" (UID: "407ec7cd-8278-4a86-b8ff-0fb8500b531a"). InnerVolumeSpecName "kube-api-access-6dqk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:31.839525 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.839422    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd4230d-82fc-4310-a897-744c6deb6404-kube-api-access-2s4vx\" (UniqueName: \"kubernetes.io/projected/9cd4230d-82fc-4310-a897-744c6deb6404-kube-api-access-2s4vx\") pod \"9cd4230d-82fc-4310-a897-744c6deb6404\" (UID: \"9cd4230d-82fc-4310-a897-744c6deb6404\") "
Oct 30 23:04:31.839525 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.839503    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd4230d-82fc-4310-a897-744c6deb6404-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd4230d-82fc-4310-a897-744c6deb6404-plugins-config-volume\") pod \"9cd4230d-82fc-4310-a897-744c6deb6404\" (UID: \"9cd4230d-82fc-4310-a897-744c6deb6404\") "
Oct 30 23:04:31.839893 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.839870    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dqk5\" (UniqueName: \"kubernetes.io/projected/407ec7cd-8278-4a86-b8ff-0fb8500b531a-kube-api-access-6dqk5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:31.840018 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.839942    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd4230d-82fc-4310-a897-744c6deb6404-plugins-config-volume" pod "9cd4230d-82fc-4310-a897-744c6deb6404" (UID: "9cd4230d-82fc-4310-a897-744c6deb6404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:31.841822 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.841800    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd4230d-82fc-4310-a897-744c6deb6404-kube-api-access-2s4vx" pod "9cd4230d-82fc-4310-a897-744c6deb6404" (UID: "9cd4230d-82fc-4310-a897-744c6deb6404"). InnerVolumeSpecName "kube-api-access-2s4vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:31.935001 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.934964    4581 generic.go:356] "Generic (PLEG): container finished" podID="9cd4230d-82fc-4310-a897-744c6deb6404" containerID="a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0" exitCode=0
Oct 30 23:04:31.935463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.935017    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"
Oct 30 23:04:31.935463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.935048    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r" event={"ID":"9cd4230d-82fc-4310-a897-744c6deb6404","Type":"ContainerDied","Data":"a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0"}
Oct 30 23:04:31.935463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.935095    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r" event={"ID":"9cd4230d-82fc-4310-a897-744c6deb6404","Type":"ContainerDied","Data":"648a5dee217c07065f44cd2b893e01d0f446a383bf4019f28faee3878c4e9fe4"}
Oct 30 23:04:31.935463 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.935122    4581 scope.go:122] "RemoveContainer" containerID="a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0"
Oct 30 23:04:31.937152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.937125    4581 generic.go:356] "Generic (PLEG): container finished" podID="cb6495fe-1800-42a5-9b56-04c4b5a74745" containerID="38dc3567ea9a92df8856e6763bcabf3d5955de0dacd6b0e0629cbee6b1a20e60" exitCode=0
Oct 30 23:04:31.937335 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.937205    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn" event={"ID":"cb6495fe-1800-42a5-9b56-04c4b5a74745","Type":"ContainerDied","Data":"38dc3567ea9a92df8856e6763bcabf3d5955de0dacd6b0e0629cbee6b1a20e60"}
Oct 30 23:04:31.939182 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.939153    4581 generic.go:356] "Generic (PLEG): container finished" podID="88c35483-4333-4938-9d5f-ed2e7330b588" containerID="b603428e49d4020febacc5e18e2edf1ff8131dc64b8e0524cdf257ff1043ff7c" exitCode=0
Oct 30 23:04:31.939305 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.939280    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm" event={"ID":"88c35483-4333-4938-9d5f-ed2e7330b588","Type":"ContainerDied","Data":"b603428e49d4020febacc5e18e2edf1ff8131dc64b8e0524cdf257ff1043ff7c"}
Oct 30 23:04:31.940684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.940660    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s4vx\" (UniqueName: \"kubernetes.io/projected/9cd4230d-82fc-4310-a897-744c6deb6404-kube-api-access-2s4vx\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:31.940794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.940689    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd4230d-82fc-4310-a897-744c6deb6404-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:31.941024 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.941001    4581 generic.go:356] "Generic (PLEG): container finished" podID="407ec7cd-8278-4a86-b8ff-0fb8500b531a" containerID="1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097" exitCode=0
Oct 30 23:04:31.941067 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.941036    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"
Oct 30 23:04:31.941067 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.941045    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29" event={"ID":"407ec7cd-8278-4a86-b8ff-0fb8500b531a","Type":"ContainerDied","Data":"1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097"}
Oct 30 23:04:31.941130 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.941087    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29" event={"ID":"407ec7cd-8278-4a86-b8ff-0fb8500b531a","Type":"ContainerDied","Data":"df085f9708885646d9a2767a1c5ac627833ec656a6a4edbc7b41a6910b1fad1a"}
Oct 30 23:04:31.950425 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.950372    4581 scope.go:122] "RemoveContainer" containerID="a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0"
Oct 30 23:04:31.950758 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:31.950736    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0\": container with ID starting with a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0 not found: ID does not exist" containerID="a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0"
Oct 30 23:04:31.950840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.950766    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0"} err="failed to get container status \"a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0\": rpc error: code = NotFound desc = could not find container \"a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0\": container with ID starting with a3469f41b94c899ce6f4f451b2d69a0d000b2430afe769a8973194c16949aec0 not found: ID does not exist"
Oct 30 23:04:31.950840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.950785    4581 scope.go:122] "RemoveContainer" containerID="1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097"
Oct 30 23:04:31.964854 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.964825    4581 scope.go:122] "RemoveContainer" containerID="1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097"
Oct 30 23:04:31.965202 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:31.965174    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097\": container with ID starting with 1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097 not found: ID does not exist" containerID="1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097"
Oct 30 23:04:31.965275 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.965242    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097"} err="failed to get container status \"1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097\": rpc error: code = NotFound desc = could not find container \"1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097\": container with ID starting with 1e89d232099b45120b77c5d576939eb04b11c62860991ff25c9156505acd6097 not found: ID does not exist"
Oct 30 23:04:31.995530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:31.995495    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"]
Oct 30 23:04:32.006885 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:32.006849    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-23aab2-67f75c4687-2wq29"]
Oct 30 23:04:32.100266 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:32.100162    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407ec7cd-8278-4a86-b8ff-0fb8500b531a" path="/var/lib/kubelet/pods/407ec7cd-8278-4a86-b8ff-0fb8500b531a/volumes"
Oct 30 23:04:32.109863 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:32.109837    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"]
Oct 30 23:04:32.113900 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:32.113868    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-23aab2-ff485dcf-pzd2r"]
Oct 30 23:04:32.401681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:32.401627    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m" podUID="709c3ea4-f775-4be6-8d58-4bcaa7b53277" containerName="payload-processing" containerID="cri-o://43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40" gracePeriod=25
Oct 30 23:04:32.431335 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:32.431291    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj" podUID="7c726e47-afbb-478a-a30e-91f5177e570c" containerName="payload-pre-processing" containerID="cri-o://23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215" gracePeriod=25
Oct 30 23:04:33.104440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.104412    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:33.140662 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.140634    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:33.158606 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.158568    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-xks-ca\") pod \"cb6495fe-1800-42a5-9b56-04c4b5a74745\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") "
Oct 30 23:04:33.158799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.158608    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-openshift-ca\") pod \"cb6495fe-1800-42a5-9b56-04c4b5a74745\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") "
Oct 30 23:04:33.158799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.158697    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-openshift-ca\") pod \"88c35483-4333-4938-9d5f-ed2e7330b588\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") "
Oct 30 23:04:33.158799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.158743    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c35483-4333-4938-9d5f-ed2e7330b588-kube-api-access-nwd7f\" (UniqueName: \"kubernetes.io/projected/88c35483-4333-4938-9d5f-ed2e7330b588-kube-api-access-nwd7f\") pod \"88c35483-4333-4938-9d5f-ed2e7330b588\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") "
Oct 30 23:04:33.158799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.158761    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6495fe-1800-42a5-9b56-04c4b5a74745-kube-api-access-8bhqx\" (UniqueName: \"kubernetes.io/projected/cb6495fe-1800-42a5-9b56-04c4b5a74745-kube-api-access-8bhqx\") pod \"cb6495fe-1800-42a5-9b56-04c4b5a74745\" (UID: \"cb6495fe-1800-42a5-9b56-04c4b5a74745\") "
Oct 30 23:04:33.159057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.158895    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-xks-ca\") pod \"88c35483-4333-4938-9d5f-ed2e7330b588\" (UID: \"88c35483-4333-4938-9d5f-ed2e7330b588\") "
Oct 30 23:04:33.159158 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.159130    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-openshift-ca" pod "88c35483-4333-4938-9d5f-ed2e7330b588" (UID: "88c35483-4333-4938-9d5f-ed2e7330b588"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:33.159576 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.159387    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:33.160602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.160560    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-openshift-ca" pod "cb6495fe-1800-42a5-9b56-04c4b5a74745" (UID: "cb6495fe-1800-42a5-9b56-04c4b5a74745"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:33.161496 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.161472    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c35483-4333-4938-9d5f-ed2e7330b588-kube-api-access-nwd7f" pod "88c35483-4333-4938-9d5f-ed2e7330b588" (UID: "88c35483-4333-4938-9d5f-ed2e7330b588"). InnerVolumeSpecName "kube-api-access-nwd7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:33.161607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.161550    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-xks-ca" pod "88c35483-4333-4938-9d5f-ed2e7330b588" (UID: "88c35483-4333-4938-9d5f-ed2e7330b588"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:33.165003 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.164971    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6495fe-1800-42a5-9b56-04c4b5a74745-kube-api-access-8bhqx" pod "cb6495fe-1800-42a5-9b56-04c4b5a74745" (UID: "cb6495fe-1800-42a5-9b56-04c4b5a74745"). InnerVolumeSpecName "kube-api-access-8bhqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:33.165003 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.164973    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-xks-ca" pod "cb6495fe-1800-42a5-9b56-04c4b5a74745" (UID: "cb6495fe-1800-42a5-9b56-04c4b5a74745"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:33.260480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.260398    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:33.260480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.260430    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6495fe-1800-42a5-9b56-04c4b5a74745-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:33.260480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.260440    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwd7f\" (UniqueName: \"kubernetes.io/projected/88c35483-4333-4938-9d5f-ed2e7330b588-kube-api-access-nwd7f\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:33.260480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.260451    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bhqx\" (UniqueName: \"kubernetes.io/projected/cb6495fe-1800-42a5-9b56-04c4b5a74745-kube-api-access-8bhqx\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:33.260480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.260461    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35483-4333-4938-9d5f-ed2e7330b588-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:33.965158 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.965125    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn" event={"ID":"cb6495fe-1800-42a5-9b56-04c4b5a74745","Type":"ContainerDied","Data":"f914545c9ef7674f1d9dd7ba05417716506c4f073f5144cdc353549e139e83aa"}
Oct 30 23:04:33.965349 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.965205    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f914545c9ef7674f1d9dd7ba05417716506c4f073f5144cdc353549e139e83aa"
Oct 30 23:04:33.965349 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.965132    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"
Oct 30 23:04:33.966678 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.966660    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"
Oct 30 23:04:33.966831 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.966680    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm" event={"ID":"88c35483-4333-4938-9d5f-ed2e7330b588","Type":"ContainerDied","Data":"e08e9b506fe478473f84033c11d31d101ad34ea7bfb5ce5ba941bd24b1a77c9d"}
Oct 30 23:04:33.966831 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:33.966722    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e08e9b506fe478473f84033c11d31d101ad34ea7bfb5ce5ba941bd24b1a77c9d"
Oct 30 23:04:34.096516 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:34.096484    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd4230d-82fc-4310-a897-744c6deb6404" path="/var/lib/kubelet/pods/9cd4230d-82fc-4310-a897-744c6deb6404/volumes"
Oct 30 23:04:35.059377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:35.059339    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"]
Oct 30 23:04:35.088838 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:35.088808    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"]
Oct 30 23:04:36.174309 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.173698    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"]
Oct 30 23:04:36.177391 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.177364    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bff950-zwwwc"]
Oct 30 23:04:36.250972 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.250934    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"]
Oct 30 23:04:36.251276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.251241    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy" containerID="cri-o://3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999" gracePeriod=30
Oct 30 23:04:36.575146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.575045    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" containerID="cri-o://cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb" gracePeriod=45
Oct 30 23:04:36.588942 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.588862    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" containerID="cri-o://7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462" gracePeriod=45
Oct 30 23:04:36.700185 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.700137    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused"
Oct 30 23:04:36.736334 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.736294    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused"
Oct 30 23:04:36.922054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.922017    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 23:04:36.925762 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:36.925738    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.005941    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-tmp-dir\") pod \"156793bd-b877-497a-8c48-7a1f982a5e78\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.005989    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156793bd-b877-497a-8c48-7a1f982a5e78-tls-certs\" (UniqueName: \"kubernetes.io/secret/156793bd-b877-497a-8c48-7a1f982a5e78-tls-certs\") pod \"156793bd-b877-497a-8c48-7a1f982a5e78\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006025    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-home\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-home\") pod \"156793bd-b877-497a-8c48-7a1f982a5e78\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006076    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-dshm\") pod \"156793bd-b877-497a-8c48-7a1f982a5e78\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006093    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-dshm\") pod \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006113    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bf79be-6e7c-4780-9d50-5d13d5d57bae-kube-api-access-6cpkc\" (UniqueName: \"kubernetes.io/projected/46bf79be-6e7c-4780-9d50-5d13d5d57bae-kube-api-access-6cpkc\") pod \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006136    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-home\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-home\") pod \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006156    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-model-cache\") pod \"156793bd-b877-497a-8c48-7a1f982a5e78\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006188    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-model-cache\") pod \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006205    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156793bd-b877-497a-8c48-7a1f982a5e78-kube-api-access-plt6r\" (UniqueName: \"kubernetes.io/projected/156793bd-b877-497a-8c48-7a1f982a5e78-kube-api-access-plt6r\") pod \"156793bd-b877-497a-8c48-7a1f982a5e78\" (UID: \"156793bd-b877-497a-8c48-7a1f982a5e78\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006253    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tls-certs\") pod \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006280    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tmp-dir\") pod \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\" (UID: \"46bf79be-6e7c-4780-9d50-5d13d5d57bae\") "
Oct 30 23:04:37.007288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.006893    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tmp-dir" pod "46bf79be-6e7c-4780-9d50-5d13d5d57bae" (UID: "46bf79be-6e7c-4780-9d50-5d13d5d57bae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:37.008178 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.007414    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-model-cache" pod "46bf79be-6e7c-4780-9d50-5d13d5d57bae" (UID: "46bf79be-6e7c-4780-9d50-5d13d5d57bae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:37.008178 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.007435    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-home" pod "156793bd-b877-497a-8c48-7a1f982a5e78" (UID: "156793bd-b877-497a-8c48-7a1f982a5e78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:37.008178 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.007599    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-model-cache" pod "156793bd-b877-497a-8c48-7a1f982a5e78" (UID: "156793bd-b877-497a-8c48-7a1f982a5e78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:37.008178 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.007861    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-tmp-dir" pod "156793bd-b877-497a-8c48-7a1f982a5e78" (UID: "156793bd-b877-497a-8c48-7a1f982a5e78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:37.008178 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.007866    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-home" pod "46bf79be-6e7c-4780-9d50-5d13d5d57bae" (UID: "46bf79be-6e7c-4780-9d50-5d13d5d57bae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:37.012338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.010685    4581 generic.go:356] "Generic (PLEG): container finished" podID="156793bd-b877-497a-8c48-7a1f982a5e78" containerID="cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb" exitCode=0
Oct 30 23:04:37.012338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.010717    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-dshm" pod "156793bd-b877-497a-8c48-7a1f982a5e78" (UID: "156793bd-b877-497a-8c48-7a1f982a5e78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:37.012338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.010824    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"
Oct 30 23:04:37.012338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.010933    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" event={"ID":"156793bd-b877-497a-8c48-7a1f982a5e78","Type":"ContainerDied","Data":"cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb"}
Oct 30 23:04:37.012338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.010968    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq" event={"ID":"156793bd-b877-497a-8c48-7a1f982a5e78","Type":"ContainerDied","Data":"24b6bf99c3be18fc542c832499f5dafac3f05d52fa5ac83440800845264c618a"}
Oct 30 23:04:37.012338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.010990    4581 scope.go:122] "RemoveContainer" containerID="cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb"
Oct 30 23:04:37.012338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.011172    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156793bd-b877-497a-8c48-7a1f982a5e78-tls-certs" pod "156793bd-b877-497a-8c48-7a1f982a5e78" (UID: "156793bd-b877-497a-8c48-7a1f982a5e78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:37.013451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.013425    4581 generic.go:356] "Generic (PLEG): container finished" podID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerID="7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462" exitCode=0
Oct 30 23:04:37.013589 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.013545    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" event={"ID":"46bf79be-6e7c-4780-9d50-5d13d5d57bae","Type":"ContainerDied","Data":"7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462"}
Oct 30 23:04:37.013589 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.013573    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7" event={"ID":"46bf79be-6e7c-4780-9d50-5d13d5d57bae","Type":"ContainerDied","Data":"08637c82cddc68dd2aeea753c7f101927c47080bebf90f8b3299d24c6d52aa31"}
Oct 30 23:04:37.013715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.013618    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"
Oct 30 23:04:37.014038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.014010    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bf79be-6e7c-4780-9d50-5d13d5d57bae-kube-api-access-6cpkc" pod "46bf79be-6e7c-4780-9d50-5d13d5d57bae" (UID: "46bf79be-6e7c-4780-9d50-5d13d5d57bae"). InnerVolumeSpecName "kube-api-access-6cpkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:37.014157 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.014097    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156793bd-b877-497a-8c48-7a1f982a5e78-kube-api-access-plt6r" pod "156793bd-b877-497a-8c48-7a1f982a5e78" (UID: "156793bd-b877-497a-8c48-7a1f982a5e78"). InnerVolumeSpecName "kube-api-access-plt6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:37.014301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.014279    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-dshm" pod "46bf79be-6e7c-4780-9d50-5d13d5d57bae" (UID: "46bf79be-6e7c-4780-9d50-5d13d5d57bae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:37.014510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.014482    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tls-certs" pod "46bf79be-6e7c-4780-9d50-5d13d5d57bae" (UID: "46bf79be-6e7c-4780-9d50-5d13d5d57bae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:37.035033 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.035005    4581 scope.go:122] "RemoveContainer" containerID="cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb"
Oct 30 23:04:37.035528 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:37.035500    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb\": container with ID starting with cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb not found: ID does not exist" containerID="cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb"
Oct 30 23:04:37.035703 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.035667    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb"} err="failed to get container status \"cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb\": rpc error: code = NotFound desc = could not find container \"cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb\": container with ID starting with cab3f5aac2873fe5315d029597bbf881aff9ceb003e850ab899b9af7f4837fdb not found: ID does not exist"
Oct 30 23:04:37.035843 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.035706    4581 scope.go:122] "RemoveContainer" containerID="7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462"
Oct 30 23:04:37.048056 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.048032    4581 scope.go:122] "RemoveContainer" containerID="7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462"
Oct 30 23:04:37.048412 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:37.048393    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462\": container with ID starting with 7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462 not found: ID does not exist" containerID="7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462"
Oct 30 23:04:37.048466 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.048423    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462"} err="failed to get container status \"7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462\": rpc error: code = NotFound desc = could not find container \"7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462\": container with ID starting with 7b9996ccac2d280a5608eb92d62b95e6f89389eec5d04633be4f6394ee7a7462 not found: ID does not exist"
Oct 30 23:04:37.107899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107867    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.107899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107894    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.107899 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107905    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107914    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plt6r\" (UniqueName: \"kubernetes.io/projected/156793bd-b877-497a-8c48-7a1f982a5e78-kube-api-access-plt6r\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107924    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107949    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107957    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107964    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156793bd-b877-497a-8c48-7a1f982a5e78-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107971    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107978    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156793bd-b877-497a-8c48-7a1f982a5e78-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107985    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf79be-6e7c-4780-9d50-5d13d5d57bae-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.108140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.107992    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpkc\" (UniqueName: \"kubernetes.io/projected/46bf79be-6e7c-4780-9d50-5d13d5d57bae-kube-api-access-6cpkc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.356306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.356270    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"]
Oct 30 23:04:37.361072 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.361045    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-421583/premium-w2-421583-kserve-655b46c7dc-hkdcq"]
Oct 30 23:04:37.386467 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.386432    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"]
Oct 30 23:04:37.390730 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:37.390696    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-421583/simulator-w2-421583-kserve-6c8f496b58-s75q7"]
Oct 30 23:04:38.098465 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:38.098422    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" path="/var/lib/kubelet/pods/156793bd-b877-497a-8c48-7a1f982a5e78/volumes"
Oct 30 23:04:38.098929 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:38.098908    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3" path="/var/lib/kubelet/pods/1b45d7fd-d2a8-48bd-a2ef-feeb3b9ddea3/volumes"
Oct 30 23:04:38.099388 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:38.099368    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" path="/var/lib/kubelet/pods/46bf79be-6e7c-4780-9d50-5d13d5d57bae/volumes"
Oct 30 23:04:40.060633 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.060598    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd" podUID="6d0b8bab-0940-4b75-b95b-f5a167ac19bf" containerName="payload-processing" containerID="cri-o://2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777" gracePeriod=25
Oct 30 23:04:40.089080 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.089050    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v" podUID="9adb45cf-8bab-4ea3-bb70-9af14c11453d" containerName="payload-pre-processing" containerID="cri-o://8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af" gracePeriod=25
Oct 30 23:04:40.352680 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.352652    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 23:04:40.379095 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.379060    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 23:04:40.447870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.447840    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-kube-api-access-hj4zd\" (UniqueName: \"kubernetes.io/projected/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-kube-api-access-hj4zd\") pod \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\" (UID: \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\") "
Oct 30 23:04:40.448047 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.447922    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-plugins-config-volume\") pod \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\" (UID: \"6d0b8bab-0940-4b75-b95b-f5a167ac19bf\") "
Oct 30 23:04:40.448093 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.448039    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9adb45cf-8bab-4ea3-bb70-9af14c11453d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adb45cf-8bab-4ea3-bb70-9af14c11453d-plugins-config-volume\") pod \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\" (UID: \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\") "
Oct 30 23:04:40.448093 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.448078    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9adb45cf-8bab-4ea3-bb70-9af14c11453d-kube-api-access-crgsw\" (UniqueName: \"kubernetes.io/projected/9adb45cf-8bab-4ea3-bb70-9af14c11453d-kube-api-access-crgsw\") pod \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\" (UID: \"9adb45cf-8bab-4ea3-bb70-9af14c11453d\") "
Oct 30 23:04:40.448485 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.448451    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-plugins-config-volume" pod "6d0b8bab-0940-4b75-b95b-f5a167ac19bf" (UID: "6d0b8bab-0940-4b75-b95b-f5a167ac19bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:40.448604 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.448485    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adb45cf-8bab-4ea3-bb70-9af14c11453d-plugins-config-volume" pod "9adb45cf-8bab-4ea3-bb70-9af14c11453d" (UID: "9adb45cf-8bab-4ea3-bb70-9af14c11453d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:40.450480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.450448    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adb45cf-8bab-4ea3-bb70-9af14c11453d-kube-api-access-crgsw" pod "9adb45cf-8bab-4ea3-bb70-9af14c11453d" (UID: "9adb45cf-8bab-4ea3-bb70-9af14c11453d"). InnerVolumeSpecName "kube-api-access-crgsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:40.450919 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.450895    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-kube-api-access-hj4zd" pod "6d0b8bab-0940-4b75-b95b-f5a167ac19bf" (UID: "6d0b8bab-0940-4b75-b95b-f5a167ac19bf"). InnerVolumeSpecName "kube-api-access-hj4zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:40.550014 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.549977    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:40.550014 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.550010    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9adb45cf-8bab-4ea3-bb70-9af14c11453d-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:40.550248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.550023    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgsw\" (UniqueName: \"kubernetes.io/projected/9adb45cf-8bab-4ea3-bb70-9af14c11453d-kube-api-access-crgsw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:40.550248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:40.550039    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj4zd\" (UniqueName: \"kubernetes.io/projected/6d0b8bab-0940-4b75-b95b-f5a167ac19bf-kube-api-access-hj4zd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:40.817478 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:40.817435    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:04:41.053909 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.053869    4581 generic.go:356] "Generic (PLEG): container finished" podID="9adb45cf-8bab-4ea3-bb70-9af14c11453d" containerID="8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af" exitCode=0
Oct 30 23:04:41.054100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.053937    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"
Oct 30 23:04:41.054100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.053955    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v" event={"ID":"9adb45cf-8bab-4ea3-bb70-9af14c11453d","Type":"ContainerDied","Data":"8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af"}
Oct 30 23:04:41.054100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.054045    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v" event={"ID":"9adb45cf-8bab-4ea3-bb70-9af14c11453d","Type":"ContainerDied","Data":"8d884a05d835ee1a936fa7de6262bd0f974aa82036ca0a1a8b6a275549ac49a7"}
Oct 30 23:04:41.054100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.054075    4581 scope.go:122] "RemoveContainer" containerID="8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af"
Oct 30 23:04:41.055885 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.055858    4581 generic.go:356] "Generic (PLEG): container finished" podID="6d0b8bab-0940-4b75-b95b-f5a167ac19bf" containerID="2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777" exitCode=0
Oct 30 23:04:41.056310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.056283    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd" event={"ID":"6d0b8bab-0940-4b75-b95b-f5a167ac19bf","Type":"ContainerDied","Data":"2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777"}
Oct 30 23:04:41.056454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.056331    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"
Oct 30 23:04:41.056454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.056338    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd" event={"ID":"6d0b8bab-0940-4b75-b95b-f5a167ac19bf","Type":"ContainerDied","Data":"93a4056511d1b7b8d6857fd126bc829e3951e68bc633ae1f5540f8f02b14f23e"}
Oct 30 23:04:41.068088 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.067860    4581 scope.go:122] "RemoveContainer" containerID="8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af"
Oct 30 23:04:41.068435 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:41.068198    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af\": container with ID starting with 8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af not found: ID does not exist" containerID="8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af"
Oct 30 23:04:41.068435 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.068264    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af"} err="failed to get container status \"8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af\": rpc error: code = NotFound desc = could not find container \"8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af\": container with ID starting with 8c6f49c4d3bab6333a223d4e27dc795690343d4a83aeb2e639dce7fd5dac76af not found: ID does not exist"
Oct 30 23:04:41.068435 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.068285    4581 scope.go:122] "RemoveContainer" containerID="2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777"
Oct 30 23:04:41.084256 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.084201    4581 scope.go:122] "RemoveContainer" containerID="2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777"
Oct 30 23:04:41.084557 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:41.084537    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777\": container with ID starting with 2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777 not found: ID does not exist" containerID="2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777"
Oct 30 23:04:41.084626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.084566    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777"} err="failed to get container status \"2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777\": rpc error: code = NotFound desc = could not find container \"2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777\": container with ID starting with 2f9de1e6fc67b3e64dcc7add8b9f0c7b8cd85b1cf9942d8d517c8cb9ece2b777 not found: ID does not exist"
Oct 30 23:04:41.112338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.112306    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"]
Oct 30 23:04:41.116313 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.116291    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2b1e88-6f4479c569-mltjd"]
Oct 30 23:04:41.151243 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.151196    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"]
Oct 30 23:04:41.155344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.155322    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2b1e88-65c766876c-4qz4v"]
Oct 30 23:04:41.517605 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.517573    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 23:04:41.561825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.561774    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-certs\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.561825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.561825    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-envoy\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.562194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.561869    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-data\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.562194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.561904    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-kube-api-access-rzxtg\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-kube-api-access-rzxtg\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.562194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.561957    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-credential-socket\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.562194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.562041    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-podinfo\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.562194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.562095    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-socket\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.563030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.562695    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-credential-socket" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:41.563030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.562765    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istiod-ca-cert\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.563030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.562855    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-token\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-token\") pod \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\" (UID: \"c401bcec-d339-41e0-8a64-ec96d4fdc5eb\") "
Oct 30 23:04:41.563030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.562855    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-data" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:41.563030 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.562964    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-certs" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:41.563520 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.563339    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istiod-ca-cert" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:41.563520 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.563395    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-socket" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:41.567810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.565073    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.567810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.565102    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.567810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.565115    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.567810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.565133    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.567810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.565148    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.567810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.566572    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-podinfo" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:41.567810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.567015    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-envoy" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:41.567810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.567658    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-kube-api-access-rzxtg" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "kube-api-access-rzxtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:41.569384 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.569356    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-token" pod "c401bcec-d339-41e0-8a64-ec96d4fdc5eb" (UID: "c401bcec-d339-41e0-8a64-ec96d4fdc5eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:41.605669 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.605621    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj" podUID="7c726e47-afbb-478a-a30e-91f5177e570c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:41.605819 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.605702    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m" podUID="709c3ea4-f775-4be6-8d58-4bcaa7b53277" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:41.667179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.667091    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.667179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.667121    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.667179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.667132    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:41.667179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:41.667141    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzxtg\" (UniqueName: \"kubernetes.io/projected/c401bcec-d339-41e0-8a64-ec96d4fdc5eb-kube-api-access-rzxtg\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:42.021064 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.020958    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"]
Oct 30 23:04:42.026763 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.026727    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-421583-wdw5x"]
Oct 30 23:04:42.070419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.070386    4581 generic.go:356] "Generic (PLEG): container finished" podID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerID="3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999" exitCode=0
Oct 30 23:04:42.070932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.070452    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"
Oct 30 23:04:42.070932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.070479    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" event={"ID":"c401bcec-d339-41e0-8a64-ec96d4fdc5eb","Type":"ContainerDied","Data":"3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999"}
Oct 30 23:04:42.070932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.070545    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc" event={"ID":"c401bcec-d339-41e0-8a64-ec96d4fdc5eb","Type":"ContainerDied","Data":"c50621bfce086e2510147f24c77e1a292938d54548d74112177e280c5dd66406"}
Oct 30 23:04:42.070932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.070579    4581 scope.go:122] "RemoveContainer" containerID="3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999"
Oct 30 23:04:42.091557 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.089076    4581 scope.go:122] "RemoveContainer" containerID="91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f"
Oct 30 23:04:42.098906 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.098870    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0b8bab-0940-4b75-b95b-f5a167ac19bf" path="/var/lib/kubelet/pods/6d0b8bab-0940-4b75-b95b-f5a167ac19bf/volumes"
Oct 30 23:04:42.099533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.099513    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9adb45cf-8bab-4ea3-bb70-9af14c11453d" path="/var/lib/kubelet/pods/9adb45cf-8bab-4ea3-bb70-9af14c11453d/volumes"
Oct 30 23:04:42.100040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.100025    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff27a071-df0b-4285-a2be-bbaef08bebe6" path="/var/lib/kubelet/pods/ff27a071-df0b-4285-a2be-bbaef08bebe6/volumes"
Oct 30 23:04:42.102968 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.102945    4581 scope.go:122] "RemoveContainer" containerID="3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999"
Oct 30 23:04:42.103297 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:42.103278    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999\": container with ID starting with 3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999 not found: ID does not exist" containerID="3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999"
Oct 30 23:04:42.103364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.103304    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999"} err="failed to get container status \"3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999\": rpc error: code = NotFound desc = could not find container \"3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999\": container with ID starting with 3bddf26bb6df212d9288d1ecd23f74aa8484a2aec93d42b2f6cbaf27ec6b9999 not found: ID does not exist"
Oct 30 23:04:42.103364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.103323    4581 scope.go:122] "RemoveContainer" containerID="91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f"
Oct 30 23:04:42.103604 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:42.103585    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f\": container with ID starting with 91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f not found: ID does not exist" containerID="91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f"
Oct 30 23:04:42.103667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.103613    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f"} err="failed to get container status \"91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f\": rpc error: code = NotFound desc = could not find container \"91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f\": container with ID starting with 91518ab345d8bf0da61b31934f5a441b4e69e914b8b0d40dbbad71193d80576f not found: ID does not exist"
Oct 30 23:04:42.118295 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.118268    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"]
Oct 30 23:04:42.122487 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:42.122463    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-tdvdc"]
Oct 30 23:04:43.525887 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.525823    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"]
Oct 30 23:04:43.526607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526586    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9cd4230d-82fc-4310-a897-744c6deb6404" containerName="payload-pre-processing"
Oct 30 23:04:43.526663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526609    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd4230d-82fc-4310-a897-744c6deb6404" containerName="payload-pre-processing"
Oct 30 23:04:43.526663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526627    4581 cpu_manager.go:414] "containerMap: removing container" podUID="88c35483-4333-4938-9d5f-ed2e7330b588" containerName="revoke-keys"
Oct 30 23:04:43.526663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526634    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c35483-4333-4938-9d5f-ed2e7330b588" containerName="revoke-keys"
Oct 30 23:04:43.526663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526650    4581 cpu_manager.go:414] "containerMap: removing container" podUID="9adb45cf-8bab-4ea3-bb70-9af14c11453d" containerName="payload-pre-processing"
Oct 30 23:04:43.526663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526656    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9adb45cf-8bab-4ea3-bb70-9af14c11453d" containerName="payload-pre-processing"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526668    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6d0b8bab-0940-4b75-b95b-f5a167ac19bf" containerName="payload-processing"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526673    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0b8bab-0940-4b75-b95b-f5a167ac19bf" containerName="payload-processing"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526685    4581 cpu_manager.go:414] "containerMap: removing container" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526689    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526704    4581 cpu_manager.go:414] "containerMap: removing container" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526709    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526719    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526724    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526730    4581 cpu_manager.go:414] "containerMap: removing container" podUID="cb6495fe-1800-42a5-9b56-04c4b5a74745" containerName="revoke-keys"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526735    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6495fe-1800-42a5-9b56-04c4b5a74745" containerName="revoke-keys"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526743    4581 cpu_manager.go:414] "containerMap: removing container" podUID="407ec7cd-8278-4a86-b8ff-0fb8500b531a" containerName="payload-processing"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526749    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407ec7cd-8278-4a86-b8ff-0fb8500b531a" containerName="payload-processing"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526761    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526766    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526830    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6495fe-1800-42a5-9b56-04c4b5a74745" containerName="revoke-keys"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526837    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="407ec7cd-8278-4a86-b8ff-0fb8500b531a" containerName="payload-processing"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526849    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526855    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd4230d-82fc-4310-a897-744c6deb6404" containerName="payload-pre-processing"
Oct 30 23:04:43.526857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526863    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c35483-4333-4938-9d5f-ed2e7330b588" containerName="revoke-keys"
Oct 30 23:04:43.527740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526870    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="9adb45cf-8bab-4ea3-bb70-9af14c11453d" containerName="payload-pre-processing"
Oct 30 23:04:43.527740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526876    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0b8bab-0940-4b75-b95b-f5a167ac19bf" containerName="payload-processing"
Oct 30 23:04:43.527740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526882    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bf79be-6e7c-4780-9d50-5d13d5d57bae" containerName="main"
Oct 30 23:04:43.527740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526890    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="156793bd-b877-497a-8c48-7a1f982a5e78" containerName="main"
Oct 30 23:04:43.527740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.526896    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" containerName="istio-proxy"
Oct 30 23:04:43.532518 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.532492    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.553653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.553624    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"]
Oct 30 23:04:43.559323 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.559009    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:04:43.559323 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.559103    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:04:43.559323 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.559196    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:04:43.559323 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.559288    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:04:43.562949 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.562922    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"]
Oct 30 23:04:43.566905 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.566887    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.582573 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.582406    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"]
Oct 30 23:04:43.590792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.590762    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkw6h\" (UniqueName: \"kubernetes.io/projected/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-kube-api-access-fkw6h\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.590917 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.590852    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.590993 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.590964    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.655048 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.655014    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"]
Oct 30 23:04:43.659135 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.659105    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.680422 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.680398    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"]
Oct 30 23:04:43.692432 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.692407    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.692616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.692445    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.692616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.692475    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.692616 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.692523    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2gn\" (UniqueName: \"kubernetes.io/projected/399109a1-199b-4b53-b2fb-a26a8b972eb2-kube-api-access-8n2gn\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.692780 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.692667    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.692780 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.692712    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkw6h\" (UniqueName: \"kubernetes.io/projected/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-kube-api-access-fkw6h\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.693252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.693172    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.695004 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.694981    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.720541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.720496    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkw6h\" (UniqueName: \"kubernetes.io/projected/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-kube-api-access-fkw6h\") pod \"maas-api-revoke-sub-5d66fac1-5hwzg\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.748930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.748892    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"]
Oct 30 23:04:43.749255 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.749194    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy" containerID="cri-o://f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4" gracePeriod=30
Oct 30 23:04:43.755445 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.755328    4581 patch_prober.go:29] interesting pod/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:04:43.755445 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.755385    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:04:43.793402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.793317    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.793402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.793363    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.793602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.793474    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5gj\" (UniqueName: \"kubernetes.io/projected/faf4f622-4393-4170-a79e-ea0a1f0c0c93-kube-api-access-kb5gj\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.793602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.793582    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.793667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.793616    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.793667 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.793637    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2gn\" (UniqueName: \"kubernetes.io/projected/399109a1-199b-4b53-b2fb-a26a8b972eb2-kube-api-access-8n2gn\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.794388 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.794359    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.795913 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.795886    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.811047 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.811016    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2gn\" (UniqueName: \"kubernetes.io/projected/399109a1-199b-4b53-b2fb-a26a8b972eb2-kube-api-access-8n2gn\") pod \"maas-api-revoke-sub-16b74c99-p6mcl\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.847802 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.847371    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:43.881924 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.881891    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:43.894877 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.894844    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5gj\" (UniqueName: \"kubernetes.io/projected/faf4f622-4393-4170-a79e-ea0a1f0c0c93-kube-api-access-kb5gj\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.895035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.894936    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.895035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.895001    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.896028 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.896000    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.897910 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.897883    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.913599 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.913571    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5gj\" (UniqueName: \"kubernetes.io/projected/faf4f622-4393-4170-a79e-ea0a1f0c0c93-kube-api-access-kb5gj\") pod \"maas-api-revoke-sub-70972bb6-lqgzc\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:43.973498 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:43.973280    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:44.106486 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.106444    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c401bcec-d339-41e0-8a64-ec96d4fdc5eb" path="/var/lib/kubelet/pods/c401bcec-d339-41e0-8a64-ec96d4fdc5eb/volumes"
Oct 30 23:04:44.172196 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.172163    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"]
Oct 30 23:04:44.173285 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:44.173253    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a7cd7b_ba2b_4086_aa42_b1db41b0165c.slice/crio-c56e96fef6c06043d2e361d29edb4ad9b726be6df20de0c19caa7fde3d7331a3 WatchSource:0}: Error finding container c56e96fef6c06043d2e361d29edb4ad9b726be6df20de0c19caa7fde3d7331a3: Status 404 returned error can't find the container with id c56e96fef6c06043d2e361d29edb4ad9b726be6df20de0c19caa7fde3d7331a3
Oct 30 23:04:44.204409 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.204380    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"]
Oct 30 23:04:44.210507 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:44.210479    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399109a1_199b_4b53_b2fb_a26a8b972eb2.slice/crio-d5d93ff26c3851c349cb7c55733a988058bf1ac23c263c4523eed6b4de0f2ae5 WatchSource:0}: Error finding container d5d93ff26c3851c349cb7c55733a988058bf1ac23c263c4523eed6b4de0f2ae5: Status 404 returned error can't find the container with id d5d93ff26c3851c349cb7c55733a988058bf1ac23c263c4523eed6b4de0f2ae5
Oct 30 23:04:44.295912 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.295880    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"]
Oct 30 23:04:44.298393 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:04:44.298367    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf4f622_4393_4170_a79e_ea0a1f0c0c93.slice/crio-fb3348344059e1cbd3392861c6e4579f0a58f25b478bb945a3709535660bd349 WatchSource:0}: Error finding container fb3348344059e1cbd3392861c6e4579f0a58f25b478bb945a3709535660bd349: Status 404 returned error can't find the container with id fb3348344059e1cbd3392861c6e4579f0a58f25b478bb945a3709535660bd349
Oct 30 23:04:44.358800 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.358767    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main" containerID="cri-o://e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a" gracePeriod=45
Oct 30 23:04:44.611007 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.610981    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 23:04:44.706301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.706272    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-model-cache\") pod \"a9304141-f4c1-4196-8ffc-acbeb31960f7\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") "
Oct 30 23:04:44.707503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.706341    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-tmp-dir\") pod \"a9304141-f4c1-4196-8ffc-acbeb31960f7\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") "
Oct 30 23:04:44.707503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.706598    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-dshm\") pod \"a9304141-f4c1-4196-8ffc-acbeb31960f7\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") "
Oct 30 23:04:44.707503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.706651    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9304141-f4c1-4196-8ffc-acbeb31960f7-kube-api-access-cgmzh\" (UniqueName: \"kubernetes.io/projected/a9304141-f4c1-4196-8ffc-acbeb31960f7-kube-api-access-cgmzh\") pod \"a9304141-f4c1-4196-8ffc-acbeb31960f7\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") "
Oct 30 23:04:44.707503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.706727    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-home\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-home\") pod \"a9304141-f4c1-4196-8ffc-acbeb31960f7\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") "
Oct 30 23:04:44.707503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.706781    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9304141-f4c1-4196-8ffc-acbeb31960f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9304141-f4c1-4196-8ffc-acbeb31960f7-tls-certs\") pod \"a9304141-f4c1-4196-8ffc-acbeb31960f7\" (UID: \"a9304141-f4c1-4196-8ffc-acbeb31960f7\") "
Oct 30 23:04:44.707503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.706594    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-model-cache" pod "a9304141-f4c1-4196-8ffc-acbeb31960f7" (UID: "a9304141-f4c1-4196-8ffc-acbeb31960f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.707503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.707122    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-home" pod "a9304141-f4c1-4196-8ffc-acbeb31960f7" (UID: "a9304141-f4c1-4196-8ffc-acbeb31960f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.707503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.706815    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-tmp-dir" pod "a9304141-f4c1-4196-8ffc-acbeb31960f7" (UID: "a9304141-f4c1-4196-8ffc-acbeb31960f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.710290 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.710250    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9304141-f4c1-4196-8ffc-acbeb31960f7-kube-api-access-cgmzh" pod "a9304141-f4c1-4196-8ffc-acbeb31960f7" (UID: "a9304141-f4c1-4196-8ffc-acbeb31960f7"). InnerVolumeSpecName "kube-api-access-cgmzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:44.710626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.710605    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9304141-f4c1-4196-8ffc-acbeb31960f7-tls-certs" pod "a9304141-f4c1-4196-8ffc-acbeb31960f7" (UID: "a9304141-f4c1-4196-8ffc-acbeb31960f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:44.710927 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.710898    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-dshm" pod "a9304141-f4c1-4196-8ffc-acbeb31960f7" (UID: "a9304141-f4c1-4196-8ffc-acbeb31960f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.808043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.807954    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.808043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.807986    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgmzh\" (UniqueName: \"kubernetes.io/projected/a9304141-f4c1-4196-8ffc-acbeb31960f7-kube-api-access-cgmzh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.808043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.807996    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.808043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.808004    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9304141-f4c1-4196-8ffc-acbeb31960f7-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.808043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.808013    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.808043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:44.808022    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9304141-f4c1-4196-8ffc-acbeb31960f7-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:45.117798 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.117767    4581 generic.go:356] "Generic (PLEG): container finished" podID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerID="e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a" exitCode=0
Oct 30 23:04:45.117999 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.117815    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"
Oct 30 23:04:45.117999 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.117854    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" event={"ID":"a9304141-f4c1-4196-8ffc-acbeb31960f7","Type":"ContainerDied","Data":"e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a"}
Oct 30 23:04:45.117999 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.117900    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt" event={"ID":"a9304141-f4c1-4196-8ffc-acbeb31960f7","Type":"ContainerDied","Data":"17b3a4b22df56fbb0401d58bd3bea7eff90679e3400c674513aedf98f3305f3b"}
Oct 30 23:04:45.117999 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.117921    4581 scope.go:122] "RemoveContainer" containerID="e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a"
Oct 30 23:04:45.120301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.120245    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc" event={"ID":"faf4f622-4393-4170-a79e-ea0a1f0c0c93","Type":"ContainerStarted","Data":"06bd41e31e1caf811ad32efeb83e78dac2587f1dd62678075e997d496feb110f"}
Oct 30 23:04:45.120452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.120343    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc" event={"ID":"faf4f622-4393-4170-a79e-ea0a1f0c0c93","Type":"ContainerStarted","Data":"fb3348344059e1cbd3392861c6e4579f0a58f25b478bb945a3709535660bd349"}
Oct 30 23:04:45.122013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.121988    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg" event={"ID":"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c","Type":"ContainerStarted","Data":"7ec18a0b3f673a0aa546860afd22b078415f3cc77026fa9271f662431676996d"}
Oct 30 23:04:45.122146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.122031    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg" event={"ID":"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c","Type":"ContainerStarted","Data":"c56e96fef6c06043d2e361d29edb4ad9b726be6df20de0c19caa7fde3d7331a3"}
Oct 30 23:04:45.123518 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.123492    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl" event={"ID":"399109a1-199b-4b53-b2fb-a26a8b972eb2","Type":"ContainerStarted","Data":"5ad59a776e304c0d1c896d7baca9965ab5c828d58b2d1751712b4d653db6c518"}
Oct 30 23:04:45.123607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.123537    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl" event={"ID":"399109a1-199b-4b53-b2fb-a26a8b972eb2","Type":"ContainerStarted","Data":"d5d93ff26c3851c349cb7c55733a988058bf1ac23c263c4523eed6b4de0f2ae5"}
Oct 30 23:04:45.131010 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.130995    4581 scope.go:122] "RemoveContainer" containerID="e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a"
Oct 30 23:04:45.131300 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:45.131280    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a\": container with ID starting with e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a not found: ID does not exist" containerID="e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a"
Oct 30 23:04:45.131367 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.131310    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a"} err="failed to get container status \"e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a\": rpc error: code = NotFound desc = could not find container \"e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a\": container with ID starting with e73d791cb0bfead3fb4657860637017de2948e0bcdcd1ffc7329f7fc00bb112a not found: ID does not exist"
Oct 30 23:04:45.166800 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.166743    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg" podStartSLOduration=2.1667257859999998 podStartE2EDuration="2.166725786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:45.163616295 +0000 UTC m=+3469.202555182" watchObservedRunningTime="2026-10-30 23:04:45.166725786 +0000 UTC m=+3469.205664673"
Oct 30 23:04:45.196703 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.196675    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"]
Oct 30 23:04:45.201365 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.201341    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/distinct-w1-2b1e88-kserve-65c7bcf886-2ltwt"]
Oct 30 23:04:45.237057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.237009    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc" podStartSLOduration=2.236995334 podStartE2EDuration="2.236995334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:45.235621861 +0000 UTC m=+3469.274560747" watchObservedRunningTime="2026-10-30 23:04:45.236995334 +0000 UTC m=+3469.275934198"
Oct 30 23:04:45.272961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.272910    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl" podStartSLOduration=2.272891283 podStartE2EDuration="2.272891283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:45.269537197 +0000 UTC m=+3469.308476084" watchObservedRunningTime="2026-10-30 23:04:45.272891283 +0000 UTC m=+3469.311830169"
Oct 30 23:04:45.913574 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:45.913531    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main" containerID="cri-o://2de2c797ee841d46c9aa28e208cc50a9aee36ccb471c10edc125b04fe76042b9" gracePeriod=45
Oct 30 23:04:46.131553 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.130009    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" path="/var/lib/kubelet/pods/a9304141-f4c1-4196-8ffc-acbeb31960f7/volumes"
Oct 30 23:04:46.142291 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.142249    4581 generic.go:356] "Generic (PLEG): container finished" podID="6592923b-e495-48d3-8657-39df0b3db1d9" containerID="2de2c797ee841d46c9aa28e208cc50a9aee36ccb471c10edc125b04fe76042b9" exitCode=0
Oct 30 23:04:46.142500 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.142402    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" event={"ID":"6592923b-e495-48d3-8657-39df0b3db1d9","Type":"ContainerDied","Data":"2de2c797ee841d46c9aa28e208cc50a9aee36ccb471c10edc125b04fe76042b9"}
Oct 30 23:04:46.144641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.144608    4581 generic.go:356] "Generic (PLEG): container finished" podID="faf4f622-4393-4170-a79e-ea0a1f0c0c93" containerID="06bd41e31e1caf811ad32efeb83e78dac2587f1dd62678075e997d496feb110f" exitCode=0
Oct 30 23:04:46.144801 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.144723    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc" event={"ID":"faf4f622-4393-4170-a79e-ea0a1f0c0c93","Type":"ContainerDied","Data":"06bd41e31e1caf811ad32efeb83e78dac2587f1dd62678075e997d496feb110f"}
Oct 30 23:04:46.146791 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.146761    4581 generic.go:356] "Generic (PLEG): container finished" podID="c0a7cd7b-ba2b-4086-aa42-b1db41b0165c" containerID="7ec18a0b3f673a0aa546860afd22b078415f3cc77026fa9271f662431676996d" exitCode=0
Oct 30 23:04:46.146929 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.146865    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg" event={"ID":"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c","Type":"ContainerDied","Data":"7ec18a0b3f673a0aa546860afd22b078415f3cc77026fa9271f662431676996d"}
Oct 30 23:04:46.149287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.149256    4581 generic.go:356] "Generic (PLEG): container finished" podID="399109a1-199b-4b53-b2fb-a26a8b972eb2" containerID="5ad59a776e304c0d1c896d7baca9965ab5c828d58b2d1751712b4d653db6c518" exitCode=0
Oct 30 23:04:46.149440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.149389    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl" event={"ID":"399109a1-199b-4b53-b2fb-a26a8b972eb2","Type":"ContainerDied","Data":"5ad59a776e304c0d1c896d7baca9965ab5c828d58b2d1751712b4d653db6c518"}
Oct 30 23:04:46.200194 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.200160    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 23:04:46.324858 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.324823    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6592923b-e495-48d3-8657-39df0b3db1d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6592923b-e495-48d3-8657-39df0b3db1d9-tls-certs\") pod \"6592923b-e495-48d3-8657-39df0b3db1d9\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") "
Oct 30 23:04:46.325087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.324911    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-home\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-home\") pod \"6592923b-e495-48d3-8657-39df0b3db1d9\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") "
Oct 30 23:04:46.325087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.324997    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-tmp-dir\") pod \"6592923b-e495-48d3-8657-39df0b3db1d9\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") "
Oct 30 23:04:46.325087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325056    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-dshm\") pod \"6592923b-e495-48d3-8657-39df0b3db1d9\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") "
Oct 30 23:04:46.325087 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325082    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-model-cache\") pod \"6592923b-e495-48d3-8657-39df0b3db1d9\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") "
Oct 30 23:04:46.325384 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325152    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6592923b-e495-48d3-8657-39df0b3db1d9-kube-api-access-ch6sw\" (UniqueName: \"kubernetes.io/projected/6592923b-e495-48d3-8657-39df0b3db1d9-kube-api-access-ch6sw\") pod \"6592923b-e495-48d3-8657-39df0b3db1d9\" (UID: \"6592923b-e495-48d3-8657-39df0b3db1d9\") "
Oct 30 23:04:46.325719 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325687    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-tmp-dir" pod "6592923b-e495-48d3-8657-39df0b3db1d9" (UID: "6592923b-e495-48d3-8657-39df0b3db1d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.325813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325752    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-model-cache" pod "6592923b-e495-48d3-8657-39df0b3db1d9" (UID: "6592923b-e495-48d3-8657-39df0b3db1d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.325813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325803    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-home" pod "6592923b-e495-48d3-8657-39df0b3db1d9" (UID: "6592923b-e495-48d3-8657-39df0b3db1d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.325976 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325958    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.326059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325980    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.326059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.325995    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.328437 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.328400    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6592923b-e495-48d3-8657-39df0b3db1d9-tls-certs" pod "6592923b-e495-48d3-8657-39df0b3db1d9" (UID: "6592923b-e495-48d3-8657-39df0b3db1d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:46.328620 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.328562    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-dshm" pod "6592923b-e495-48d3-8657-39df0b3db1d9" (UID: "6592923b-e495-48d3-8657-39df0b3db1d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.328707 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.328566    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592923b-e495-48d3-8657-39df0b3db1d9-kube-api-access-ch6sw" pod "6592923b-e495-48d3-8657-39df0b3db1d9" (UID: "6592923b-e495-48d3-8657-39df0b3db1d9"). InnerVolumeSpecName "kube-api-access-ch6sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:46.426954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.426867    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592923b-e495-48d3-8657-39df0b3db1d9-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.426954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.426899    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6sw\" (UniqueName: \"kubernetes.io/projected/6592923b-e495-48d3-8657-39df0b3db1d9-kube-api-access-ch6sw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.426954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:46.426910    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592923b-e495-48d3-8657-39df0b3db1d9-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.160546 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.160515    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"
Oct 30 23:04:47.161073 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.160522    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf" event={"ID":"6592923b-e495-48d3-8657-39df0b3db1d9","Type":"ContainerDied","Data":"c0b44997aab17c0bb2c1c9fd1a13ac3d83ec6327f15cc13d71cddaed5df4d69b"}
Oct 30 23:04:47.161073 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.160618    4581 scope.go:122] "RemoveContainer" containerID="2de2c797ee841d46c9aa28e208cc50a9aee36ccb471c10edc125b04fe76042b9"
Oct 30 23:04:47.215660 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.215611    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"]
Oct 30 23:04:47.227270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.226315    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/simulator-w0-09d0d5-kserve-6f7684b647-62btf"]
Oct 30 23:04:47.342244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.342183    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:47.391891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.391860    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:47.396502 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.396474    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:47.438040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.437594    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-kube-api-access-fkw6h\" (UniqueName: \"kubernetes.io/projected/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-kube-api-access-fkw6h\") pod \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") "
Oct 30 23:04:47.438040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.437691    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-xks-ca\") pod \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") "
Oct 30 23:04:47.438040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.437735    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-openshift-ca\") pod \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\" (UID: \"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c\") "
Oct 30 23:04:47.438370 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.438114    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-openshift-ca" pod "c0a7cd7b-ba2b-4086-aa42-b1db41b0165c" (UID: "c0a7cd7b-ba2b-4086-aa42-b1db41b0165c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:47.440050 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.440014    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-xks-ca" pod "c0a7cd7b-ba2b-4086-aa42-b1db41b0165c" (UID: "c0a7cd7b-ba2b-4086-aa42-b1db41b0165c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:47.440050 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.440042    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-kube-api-access-fkw6h" pod "c0a7cd7b-ba2b-4086-aa42-b1db41b0165c" (UID: "c0a7cd7b-ba2b-4086-aa42-b1db41b0165c"). InnerVolumeSpecName "kube-api-access-fkw6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:47.538590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.538551    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399109a1-199b-4b53-b2fb-a26a8b972eb2-kube-api-access-8n2gn\" (UniqueName: \"kubernetes.io/projected/399109a1-199b-4b53-b2fb-a26a8b972eb2-kube-api-access-8n2gn\") pod \"399109a1-199b-4b53-b2fb-a26a8b972eb2\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") "
Oct 30 23:04:47.538767 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.538609    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-openshift-ca\") pod \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") "
Oct 30 23:04:47.538767 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.538640    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-xks-ca\") pod \"399109a1-199b-4b53-b2fb-a26a8b972eb2\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") "
Oct 30 23:04:47.538767 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.538694    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf4f622-4393-4170-a79e-ea0a1f0c0c93-kube-api-access-kb5gj\" (UniqueName: \"kubernetes.io/projected/faf4f622-4393-4170-a79e-ea0a1f0c0c93-kube-api-access-kb5gj\") pod \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") "
Oct 30 23:04:47.538973 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.538836    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-openshift-ca\") pod \"399109a1-199b-4b53-b2fb-a26a8b972eb2\" (UID: \"399109a1-199b-4b53-b2fb-a26a8b972eb2\") "
Oct 30 23:04:47.538973 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.538936    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-xks-ca\") pod \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\" (UID: \"faf4f622-4393-4170-a79e-ea0a1f0c0c93\") "
Oct 30 23:04:47.539106 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.538994    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-openshift-ca" pod "faf4f622-4393-4170-a79e-ea0a1f0c0c93" (UID: "faf4f622-4393-4170-a79e-ea0a1f0c0c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:47.539209 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.539186    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-openshift-ca" pod "399109a1-199b-4b53-b2fb-a26a8b972eb2" (UID: "399109a1-199b-4b53-b2fb-a26a8b972eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:47.539389 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.539371    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.539389 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.539395    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkw6h\" (UniqueName: \"kubernetes.io/projected/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-kube-api-access-fkw6h\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.539389 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.539410    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.539389 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.539424    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.539804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.539437    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.541102 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.541075    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-xks-ca" pod "399109a1-199b-4b53-b2fb-a26a8b972eb2" (UID: "399109a1-199b-4b53-b2fb-a26a8b972eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:47.541420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.541398    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-xks-ca" pod "faf4f622-4393-4170-a79e-ea0a1f0c0c93" (UID: "faf4f622-4393-4170-a79e-ea0a1f0c0c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:47.541517 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.541469    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399109a1-199b-4b53-b2fb-a26a8b972eb2-kube-api-access-8n2gn" pod "399109a1-199b-4b53-b2fb-a26a8b972eb2" (UID: "399109a1-199b-4b53-b2fb-a26a8b972eb2"). InnerVolumeSpecName "kube-api-access-8n2gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:47.541517 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.541494    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf4f622-4393-4170-a79e-ea0a1f0c0c93-kube-api-access-kb5gj" pod "faf4f622-4393-4170-a79e-ea0a1f0c0c93" (UID: "faf4f622-4393-4170-a79e-ea0a1f0c0c93"). InnerVolumeSpecName "kube-api-access-kb5gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:47.640037 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.640003    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2gn\" (UniqueName: \"kubernetes.io/projected/399109a1-199b-4b53-b2fb-a26a8b972eb2-kube-api-access-8n2gn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.640037 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.640035    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399109a1-199b-4b53-b2fb-a26a8b972eb2-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.640314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.640046    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb5gj\" (UniqueName: \"kubernetes.io/projected/faf4f622-4393-4170-a79e-ea0a1f0c0c93-kube-api-access-kb5gj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:47.640314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:47.640056    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf4f622-4393-4170-a79e-ea0a1f0c0c93-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:48.098103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.098068    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" path="/var/lib/kubelet/pods/6592923b-e495-48d3-8657-39df0b3db1d9/volumes"
Oct 30 23:04:48.173936 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.173907    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"
Oct 30 23:04:48.174481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.173900    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc" event={"ID":"faf4f622-4393-4170-a79e-ea0a1f0c0c93","Type":"ContainerDied","Data":"fb3348344059e1cbd3392861c6e4579f0a58f25b478bb945a3709535660bd349"}
Oct 30 23:04:48.174481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.174070    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb3348344059e1cbd3392861c6e4579f0a58f25b478bb945a3709535660bd349"
Oct 30 23:04:48.176523 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.176496    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg" event={"ID":"c0a7cd7b-ba2b-4086-aa42-b1db41b0165c","Type":"ContainerDied","Data":"c56e96fef6c06043d2e361d29edb4ad9b726be6df20de0c19caa7fde3d7331a3"}
Oct 30 23:04:48.176932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.176900    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c56e96fef6c06043d2e361d29edb4ad9b726be6df20de0c19caa7fde3d7331a3"
Oct 30 23:04:48.177099 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.176605    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"
Oct 30 23:04:48.179515 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.179487    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"
Oct 30 23:04:48.180053 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.180002    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl" event={"ID":"399109a1-199b-4b53-b2fb-a26a8b972eb2","Type":"ContainerDied","Data":"d5d93ff26c3851c349cb7c55733a988058bf1ac23c263c4523eed6b4de0f2ae5"}
Oct 30 23:04:48.180177 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:48.180057    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5d93ff26c3851c349cb7c55733a988058bf1ac23c263c4523eed6b4de0f2ae5"
Oct 30 23:04:49.033805 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.033782    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:04:49.159852 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.159816    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-token\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-token\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.160192 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.160152    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-socket\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.160470 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.160446    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-envoy\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.160587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.160541    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3da96f9-b10e-4413-995f-0a3dc89511d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3da96f9-b10e-4413-995f-0a3dc89511d4-istiod-ca-cert\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.160659 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.160603    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-kube-api-access-8ltdk\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-kube-api-access-8ltdk\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.160718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.160647    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-data\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.160783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.160736    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-certs\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.160840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.160821    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-credential-socket\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.161584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.160855    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-podinfo\") pod \"b3da96f9-b10e-4413-995f-0a3dc89511d4\" (UID: \"b3da96f9-b10e-4413-995f-0a3dc89511d4\") "
Oct 30 23:04:49.161584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.161192    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-certs" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:49.161584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.161408    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:49.161797 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.161651    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-data" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:49.162400 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.162118    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-credential-socket" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:49.163749 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.162712    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3da96f9-b10e-4413-995f-0a3dc89511d4-istiod-ca-cert" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:49.163749 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.163643    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-socket" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:49.164458 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.164437    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-token" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:49.164634 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.164613    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-podinfo" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:04:49.165653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.165633    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-envoy" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:49.166548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.166521    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-kube-api-access-8ltdk" pod "b3da96f9-b10e-4413-995f-0a3dc89511d4" (UID: "b3da96f9-b10e-4413-995f-0a3dc89511d4"). InnerVolumeSpecName "kube-api-access-8ltdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:49.192416 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.192380    4581 generic.go:356] "Generic (PLEG): container finished" podID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerID="f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4" exitCode=0
Oct 30 23:04:49.192864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.192429    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" event={"ID":"b3da96f9-b10e-4413-995f-0a3dc89511d4","Type":"ContainerDied","Data":"f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4"}
Oct 30 23:04:49.192864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.192449    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"
Oct 30 23:04:49.192864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.192482    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j" event={"ID":"b3da96f9-b10e-4413-995f-0a3dc89511d4","Type":"ContainerDied","Data":"16faea1ac361b5d5164b4a9a6815c16eb16c9a1e5c20c69d74e3109b04752ca0"}
Oct 30 23:04:49.192864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.192508    4581 scope.go:122] "RemoveContainer" containerID="f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4"
Oct 30 23:04:49.206076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.206054    4581 scope.go:122] "RemoveContainer" containerID="196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9"
Oct 30 23:04:49.216442 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.216422    4581 scope.go:122] "RemoveContainer" containerID="f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4"
Oct 30 23:04:49.216719 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:49.216698    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4\": container with ID starting with f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4 not found: ID does not exist" containerID="f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4"
Oct 30 23:04:49.216795 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.216728    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4"} err="failed to get container status \"f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4\": rpc error: code = NotFound desc = could not find container \"f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4\": container with ID starting with f8efd8a002a328218021d80b357bcc145ed5ba40488ebfe1c4d0f671eaacc5f4 not found: ID does not exist"
Oct 30 23:04:49.216795 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.216745    4581 scope.go:122] "RemoveContainer" containerID="196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9"
Oct 30 23:04:49.216980 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:49.216965    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9\": container with ID starting with 196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9 not found: ID does not exist" containerID="196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9"
Oct 30 23:04:49.217023 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.216984    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9"} err="failed to get container status \"196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9\": rpc error: code = NotFound desc = could not find container \"196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9\": container with ID starting with 196198d3d484afaf9b5b36588babe824879e2b68afd2e98e1888852fd9d221b9 not found: ID does not exist"
Oct 30 23:04:49.239333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.239305    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"]
Oct 30 23:04:49.243331 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.243304    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-bff950-openshift-default-5dd888d8fc-46j6j"]
Oct 30 23:04:49.262915 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.262856    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3da96f9-b10e-4413-995f-0a3dc89511d4-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:49.263035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.262922    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ltdk\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-kube-api-access-8ltdk\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:49.263035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.262940    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:49.263035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.262955    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:49.263035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.262973    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:49.263035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.262986    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:49.263035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.263001    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:49.263035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:49.263015    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3da96f9-b10e-4413-995f-0a3dc89511d4-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:50.096309 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:50.096270    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" path="/var/lib/kubelet/pods/b3da96f9-b10e-4413-995f-0a3dc89511d4/volumes"
Oct 30 23:04:51.606266 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:51.606203    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m" podUID="709c3ea4-f775-4be6-8d58-4bcaa7b53277" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:51.606860 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:51.606609    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj" podUID="7c726e47-afbb-478a-a30e-91f5177e570c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:52.707842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.707816    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 23:04:52.743538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.743513    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 23:04:52.805286 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.805205    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709c3ea4-f775-4be6-8d58-4bcaa7b53277-kube-api-access-7g27f\" (UniqueName: \"kubernetes.io/projected/709c3ea4-f775-4be6-8d58-4bcaa7b53277-kube-api-access-7g27f\") pod \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\" (UID: \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\") "
Oct 30 23:04:52.805472 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.805424    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709c3ea4-f775-4be6-8d58-4bcaa7b53277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709c3ea4-f775-4be6-8d58-4bcaa7b53277-plugins-config-volume\") pod \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\" (UID: \"709c3ea4-f775-4be6-8d58-4bcaa7b53277\") "
Oct 30 23:04:52.806005 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.805976    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709c3ea4-f775-4be6-8d58-4bcaa7b53277-plugins-config-volume" pod "709c3ea4-f775-4be6-8d58-4bcaa7b53277" (UID: "709c3ea4-f775-4be6-8d58-4bcaa7b53277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:52.807908 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.807883    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709c3ea4-f775-4be6-8d58-4bcaa7b53277-kube-api-access-7g27f" pod "709c3ea4-f775-4be6-8d58-4bcaa7b53277" (UID: "709c3ea4-f775-4be6-8d58-4bcaa7b53277"). InnerVolumeSpecName "kube-api-access-7g27f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:52.907167 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.907124    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c726e47-afbb-478a-a30e-91f5177e570c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c726e47-afbb-478a-a30e-91f5177e570c-plugins-config-volume\") pod \"7c726e47-afbb-478a-a30e-91f5177e570c\" (UID: \"7c726e47-afbb-478a-a30e-91f5177e570c\") "
Oct 30 23:04:52.907390 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.907240    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c726e47-afbb-478a-a30e-91f5177e570c-kube-api-access-ws6g8\" (UniqueName: \"kubernetes.io/projected/7c726e47-afbb-478a-a30e-91f5177e570c-kube-api-access-ws6g8\") pod \"7c726e47-afbb-478a-a30e-91f5177e570c\" (UID: \"7c726e47-afbb-478a-a30e-91f5177e570c\") "
Oct 30 23:04:52.907480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.907472    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g27f\" (UniqueName: \"kubernetes.io/projected/709c3ea4-f775-4be6-8d58-4bcaa7b53277-kube-api-access-7g27f\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:52.907540 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.907486    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709c3ea4-f775-4be6-8d58-4bcaa7b53277-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:52.907602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.907522    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c726e47-afbb-478a-a30e-91f5177e570c-plugins-config-volume" pod "7c726e47-afbb-478a-a30e-91f5177e570c" (UID: "7c726e47-afbb-478a-a30e-91f5177e570c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:52.909685 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:52.909652    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c726e47-afbb-478a-a30e-91f5177e570c-kube-api-access-ws6g8" pod "7c726e47-afbb-478a-a30e-91f5177e570c" (UID: "7c726e47-afbb-478a-a30e-91f5177e570c"). InnerVolumeSpecName "kube-api-access-ws6g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:53.009003 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.008914    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c726e47-afbb-478a-a30e-91f5177e570c-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:53.009003 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.008941    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws6g8\" (UniqueName: \"kubernetes.io/projected/7c726e47-afbb-478a-a30e-91f5177e570c-kube-api-access-ws6g8\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:04:53.238647 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.238611    4581 generic.go:356] "Generic (PLEG): container finished" podID="7c726e47-afbb-478a-a30e-91f5177e570c" containerID="23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215" exitCode=0
Oct 30 23:04:53.238889 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.238691    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"
Oct 30 23:04:53.238973 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.238888    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj" event={"ID":"7c726e47-afbb-478a-a30e-91f5177e570c","Type":"ContainerDied","Data":"23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215"}
Oct 30 23:04:53.238973 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.238943    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj" event={"ID":"7c726e47-afbb-478a-a30e-91f5177e570c","Type":"ContainerDied","Data":"aa86952ed7daac79033fbe99bf919401802410dcbb7b96b19e5d56aa23f1d611"}
Oct 30 23:04:53.239081 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.238986    4581 scope.go:122] "RemoveContainer" containerID="23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215"
Oct 30 23:04:53.241205 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.241098    4581 generic.go:356] "Generic (PLEG): container finished" podID="709c3ea4-f775-4be6-8d58-4bcaa7b53277" containerID="43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40" exitCode=0
Oct 30 23:04:53.241321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.241247    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m" event={"ID":"709c3ea4-f775-4be6-8d58-4bcaa7b53277","Type":"ContainerDied","Data":"43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40"}
Oct 30 23:04:53.241321 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.241291    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m" event={"ID":"709c3ea4-f775-4be6-8d58-4bcaa7b53277","Type":"ContainerDied","Data":"5e9bc00fc2a1a2818457b3b7adba06833bff45c9fe0742b5b7c772c8e57a2ba8"}
Oct 30 23:04:53.241436 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.241380    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"
Oct 30 23:04:53.253752 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.253731    4581 scope.go:122] "RemoveContainer" containerID="23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215"
Oct 30 23:04:53.254086 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:53.254068    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215\": container with ID starting with 23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215 not found: ID does not exist" containerID="23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215"
Oct 30 23:04:53.254160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.254099    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215"} err="failed to get container status \"23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215\": rpc error: code = NotFound desc = could not find container \"23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215\": container with ID starting with 23bcada64d56929b0bb95fe5316401e09568e1d82e1dedf8a599cd1448210215 not found: ID does not exist"
Oct 30 23:04:53.254160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.254126    4581 scope.go:122] "RemoveContainer" containerID="43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40"
Oct 30 23:04:53.266062 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.266043    4581 scope.go:122] "RemoveContainer" containerID="43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40"
Oct 30 23:04:53.266420 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:04:53.266397    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40\": container with ID starting with 43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40 not found: ID does not exist" containerID="43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40"
Oct 30 23:04:53.266474 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.266428    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40"} err="failed to get container status \"43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40\": rpc error: code = NotFound desc = could not find container \"43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40\": container with ID starting with 43a92986b26c078ff339ea54a496bac263905308634994767976171883766f40 not found: ID does not exist"
Oct 30 23:04:53.290261 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.290207    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"]
Oct 30 23:04:53.299073 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.299043    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-421583-bfd5dcc6c-7ztzj"]
Oct 30 23:04:53.339670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.339636    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"]
Oct 30 23:04:53.346680 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:53.346652    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-421583-7f9c569f65-xm89m"]
Oct 30 23:04:54.095753 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:54.095715    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709c3ea4-f775-4be6-8d58-4bcaa7b53277" path="/var/lib/kubelet/pods/709c3ea4-f775-4be6-8d58-4bcaa7b53277/volumes"
Oct 30 23:04:54.096301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:54.096281    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c726e47-afbb-478a-a30e-91f5177e570c" path="/var/lib/kubelet/pods/7c726e47-afbb-478a-a30e-91f5177e570c/volumes"
Oct 30 23:04:54.806716 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:54.806680    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"]
Oct 30 23:04:59.807896 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:04:59.807824    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4" podUID="d18be17e-6d08-4be0-a6df-7f4f8950214e" containerName="payload-pre-processing" containerID="cri-o://58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196" gracePeriod=25
Oct 30 23:05:02.105152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:02.105114    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4" podUID="d18be17e-6d08-4be0-a6df-7f4f8950214e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:02.972688 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:02.972649    4581 scope.go:122] "RemoveContainer" containerID="a8ee0e9f622022d3a5ba528ef1b6486d2c0985c924d71319a1f6f583c181f694"
Oct 30 23:05:02.983423 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:02.983397    4581 scope.go:122] "RemoveContainer" containerID="ce8bbdc8167d0fc51e6cb82b15087d271f19dc5e71aad7905a6e0d476c975aa3"
Oct 30 23:05:02.993864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:02.993843    4581 scope.go:122] "RemoveContainer" containerID="c5d87dad5f0dd57ee498892e4a3e052d020a7f760af87b3ba90364df81aa9a6f"
Oct 30 23:05:03.004534 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:03.004507    4581 scope.go:122] "RemoveContainer" containerID="193bdff4feab1f9986506386dc03947133b41ac64589a365e8f3b41aca42e0e3"
Oct 30 23:05:03.015105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:03.015074    4581 scope.go:122] "RemoveContainer" containerID="f0ed13c6f491faf149457f1b5b114082671c4ed695db6b9d45410db440545961"
Oct 30 23:05:03.026711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:03.026675    4581 scope.go:122] "RemoveContainer" containerID="225ca671be30c0a4d4b5efff6868a29077639724b4b5a62cbdbce77f19b8ceab"
Oct 30 23:05:03.476186 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:03.476154    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"]
Oct 30 23:05:03.485940 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:03.485907    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-09d0d5-hc2ds"]
Oct 30 23:05:04.079892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:04.079851    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"]
Oct 30 23:05:04.096545 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:04.096510    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014e36ec-128f-4ae2-b880-99b1bef435eb" path="/var/lib/kubelet/pods/014e36ec-128f-4ae2-b880-99b1bef435eb/volumes"
Oct 30 23:05:04.097150 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:04.097125    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2b1e88-qjrgw"]
Oct 30 23:05:04.764704 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:04.764668    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"]
Oct 30 23:05:04.765210 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:04.764877    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy" containerID="cri-o://0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d" gracePeriod=30
Oct 30 23:05:05.350771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.350737    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"]
Oct 30 23:05:05.351314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351296    4581 cpu_manager.go:414] "containerMap: removing container" podUID="faf4f622-4393-4170-a79e-ea0a1f0c0c93" containerName="revoke-keys"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351316    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf4f622-4393-4170-a79e-ea0a1f0c0c93" containerName="revoke-keys"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351333    4581 cpu_manager.go:414] "containerMap: removing container" podUID="709c3ea4-f775-4be6-8d58-4bcaa7b53277" containerName="payload-processing"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351338    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709c3ea4-f775-4be6-8d58-4bcaa7b53277" containerName="payload-processing"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351353    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351362    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351374    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351380    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351396    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7c726e47-afbb-478a-a30e-91f5177e570c" containerName="payload-pre-processing"
Oct 30 23:05:05.351403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351403    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c726e47-afbb-478a-a30e-91f5177e570c" containerName="payload-pre-processing"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351411    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c0a7cd7b-ba2b-4086-aa42-b1db41b0165c" containerName="revoke-keys"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351415    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a7cd7b-ba2b-4086-aa42-b1db41b0165c" containerName="revoke-keys"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351430    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351435    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351448    4581 cpu_manager.go:414] "containerMap: removing container" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351452    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351461    4581 cpu_manager.go:414] "containerMap: removing container" podUID="399109a1-199b-4b53-b2fb-a26a8b972eb2" containerName="revoke-keys"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351465    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399109a1-199b-4b53-b2fb-a26a8b972eb2" containerName="revoke-keys"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351543    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="399109a1-199b-4b53-b2fb-a26a8b972eb2" containerName="revoke-keys"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351556    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf4f622-4393-4170-a79e-ea0a1f0c0c93" containerName="revoke-keys"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351564    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="709c3ea4-f775-4be6-8d58-4bcaa7b53277" containerName="payload-processing"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351570    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9304141-f4c1-4196-8ffc-acbeb31960f7" containerName="main"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351576    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6592923b-e495-48d3-8657-39df0b3db1d9" containerName="main"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351580    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c726e47-afbb-478a-a30e-91f5177e570c" containerName="payload-pre-processing"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351584    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a7cd7b-ba2b-4086-aa42-b1db41b0165c" containerName="revoke-keys"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351591    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy"
Oct 30 23:05:05.351751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.351597    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3da96f9-b10e-4413-995f-0a3dc89511d4" containerName="istio-proxy"
Oct 30 23:05:05.358525 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.358499    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.377450 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.377411    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"]
Oct 30 23:05:05.382260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.382235    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:05:05.382423 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.382258    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:05:05.382545 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.382300    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:05:05.382545 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.382446    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:05:05.433253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.433207    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.433432 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.433280    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.433527 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.433503    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwdf\" (UniqueName: \"kubernetes.io/projected/1468a7fd-3f9b-4899-88df-4a83749041b0-kube-api-access-kdwdf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.534151 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.534113    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwdf\" (UniqueName: \"kubernetes.io/projected/1468a7fd-3f9b-4899-88df-4a83749041b0-kube-api-access-kdwdf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.534493 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.534199    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.534493 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.534262    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.534873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.534851    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.536769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.536747    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.552199 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.552173    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwdf\" (UniqueName: \"kubernetes.io/projected/1468a7fd-3f9b-4899-88df-4a83749041b0-kube-api-access-kdwdf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.676311 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.676280    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:05.931281 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.931172    4581 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:05:05.931661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.931292    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:05:05.990519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:05.990493    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"]
Oct 30 23:05:05.992892 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:05:05.992859    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1468a7fd_3f9b_4899_88df_4a83749041b0.slice/crio-a05ad8bc119d50ca55719952685e52a6565bc481d2488b91d59702b38570eb4b WatchSource:0}: Error finding container a05ad8bc119d50ca55719952685e52a6565bc481d2488b91d59702b38570eb4b: Status 404 returned error can't find the container with id a05ad8bc119d50ca55719952685e52a6565bc481d2488b91d59702b38570eb4b
Oct 30 23:05:06.134175 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:06.134133    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ba0d7d-e3a7-4118-9b63-9cf132a47fbd" path="/var/lib/kubelet/pods/81ba0d7d-e3a7-4118-9b63-9cf132a47fbd/volumes"
Oct 30 23:05:06.368738 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:06.368700    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf" event={"ID":"1468a7fd-3f9b-4899-88df-4a83749041b0","Type":"ContainerStarted","Data":"5b4f1d515dfd55a37fbea38365ccfd397befcc04e501a0cf69d04c08af4857d9"}
Oct 30 23:05:06.368923 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:06.368767    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf" event={"ID":"1468a7fd-3f9b-4899-88df-4a83749041b0","Type":"ContainerStarted","Data":"a05ad8bc119d50ca55719952685e52a6565bc481d2488b91d59702b38570eb4b"}
Oct 30 23:05:06.412975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:06.412918    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf" podStartSLOduration=1.412903388 podStartE2EDuration="1.412903388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:06.411770228 +0000 UTC m=+3490.450709115" watchObservedRunningTime="2026-10-30 23:05:06.412903388 +0000 UTC m=+3490.451842316"
Oct 30 23:05:07.378957 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:07.378911    4581 generic.go:356] "Generic (PLEG): container finished" podID="1468a7fd-3f9b-4899-88df-4a83749041b0" containerID="5b4f1d515dfd55a37fbea38365ccfd397befcc04e501a0cf69d04c08af4857d9" exitCode=0
Oct 30 23:05:07.379573 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:07.378987    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf" event={"ID":"1468a7fd-3f9b-4899-88df-4a83749041b0","Type":"ContainerDied","Data":"5b4f1d515dfd55a37fbea38365ccfd397befcc04e501a0cf69d04c08af4857d9"}
Oct 30 23:05:08.528367 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.528339    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:08.563942 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.563903    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1468a7fd-3f9b-4899-88df-4a83749041b0-kube-api-access-kdwdf\" (UniqueName: \"kubernetes.io/projected/1468a7fd-3f9b-4899-88df-4a83749041b0-kube-api-access-kdwdf\") pod \"1468a7fd-3f9b-4899-88df-4a83749041b0\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") "
Oct 30 23:05:08.564140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.563973    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-openshift-ca\") pod \"1468a7fd-3f9b-4899-88df-4a83749041b0\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") "
Oct 30 23:05:08.564140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.564044    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-xks-ca\") pod \"1468a7fd-3f9b-4899-88df-4a83749041b0\" (UID: \"1468a7fd-3f9b-4899-88df-4a83749041b0\") "
Oct 30 23:05:08.567000 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.565206    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-openshift-ca" pod "1468a7fd-3f9b-4899-88df-4a83749041b0" (UID: "1468a7fd-3f9b-4899-88df-4a83749041b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:08.570017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.567743    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1468a7fd-3f9b-4899-88df-4a83749041b0-kube-api-access-kdwdf" pod "1468a7fd-3f9b-4899-88df-4a83749041b0" (UID: "1468a7fd-3f9b-4899-88df-4a83749041b0"). InnerVolumeSpecName "kube-api-access-kdwdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:08.587874 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.585988    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-xks-ca" pod "1468a7fd-3f9b-4899-88df-4a83749041b0" (UID: "1468a7fd-3f9b-4899-88df-4a83749041b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:08.665382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.665251    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwdf\" (UniqueName: \"kubernetes.io/projected/1468a7fd-3f9b-4899-88df-4a83749041b0-kube-api-access-kdwdf\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:08.665382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.665289    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:08.665382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:08.665300    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1468a7fd-3f9b-4899-88df-4a83749041b0-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:09.396681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:09.396642    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"
Oct 30 23:05:09.396681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:09.396668    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf" event={"ID":"1468a7fd-3f9b-4899-88df-4a83749041b0","Type":"ContainerDied","Data":"a05ad8bc119d50ca55719952685e52a6565bc481d2488b91d59702b38570eb4b"}
Oct 30 23:05:09.396961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:09.396725    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a05ad8bc119d50ca55719952685e52a6565bc481d2488b91d59702b38570eb4b"
Oct 30 23:05:10.029801 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.029775    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 23:05:10.079472 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079374    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-socket\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.079472 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079436    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-envoy\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.079719 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079483    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-data\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.079719 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079517    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-certs\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.079719 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079567    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-credential-socket\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.079719 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079603    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4483810-5f34-443d-937c-9e7354fa50ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4483810-5f34-443d-937c-9e7354fa50ef-istiod-ca-cert\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.079719 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079683    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-kube-api-access-nz4f5\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-kube-api-access-nz4f5\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.080013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079747    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-socket" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.080013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079808    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-istio-token\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-istio-token\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.080013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079883    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4483810-5f34-443d-937c-9e7354fa50ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4483810-5f34-443d-937c-9e7354fa50ef-istio-podinfo\") pod \"c4483810-5f34-443d-937c-9e7354fa50ef\" (UID: \"c4483810-5f34-443d-937c-9e7354fa50ef\") "
Oct 30 23:05:10.080013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.079888    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-data" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.080259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.080075    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4483810-5f34-443d-937c-9e7354fa50ef-istiod-ca-cert" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:10.080259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.080209    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-credential-socket" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.080449 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.080289    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-certs" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.080511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.080458    4581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.080511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.080480    4581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-data\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.080511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.080496    4581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-workload-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.080671 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.080562    4581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-credential-socket\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.080671 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.080577    4581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4483810-5f34-443d-937c-9e7354fa50ef-istiod-ca-cert\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.082318 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.082281    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-istio-token" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:10.082591 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.082564    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-envoy" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.082748 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.082718    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4483810-5f34-443d-937c-9e7354fa50ef-istio-podinfo" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:05:10.083062 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.083028    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-kube-api-access-nz4f5" pod "c4483810-5f34-443d-937c-9e7354fa50ef" (UID: "c4483810-5f34-443d-937c-9e7354fa50ef"). InnerVolumeSpecName "kube-api-access-nz4f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:10.181754 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.181722    4581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-istio-token\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.181754 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.181750    4581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4483810-5f34-443d-937c-9e7354fa50ef-istio-podinfo\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.181754 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.181759    4581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4483810-5f34-443d-937c-9e7354fa50ef-istio-envoy\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.182013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.181768    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz4f5\" (UniqueName: \"kubernetes.io/projected/c4483810-5f34-443d-937c-9e7354fa50ef-kube-api-access-nz4f5\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.405390 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.405351    4581 generic.go:356] "Generic (PLEG): container finished" podID="c4483810-5f34-443d-937c-9e7354fa50ef" containerID="0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d" exitCode=0
Oct 30 23:05:10.405595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.405407    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"
Oct 30 23:05:10.405595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.405406    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" event={"ID":"c4483810-5f34-443d-937c-9e7354fa50ef","Type":"ContainerDied","Data":"0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d"}
Oct 30 23:05:10.405595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.405523    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb" event={"ID":"c4483810-5f34-443d-937c-9e7354fa50ef","Type":"ContainerDied","Data":"ef2e4299dea0340ac065865c09a10af0e490e536e34f5bf21593548a6830a73b"}
Oct 30 23:05:10.405595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.405552    4581 scope.go:122] "RemoveContainer" containerID="0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d"
Oct 30 23:05:10.416843 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.416821    4581 scope.go:122] "RemoveContainer" containerID="0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d"
Oct 30 23:05:10.417172 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:10.417145    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d\": container with ID starting with 0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d not found: ID does not exist" containerID="0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d"
Oct 30 23:05:10.417287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.417186    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d"} err="failed to get container status \"0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d\": rpc error: code = NotFound desc = could not find container \"0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d\": container with ID starting with 0d7ed5dfe33858ab6ab42995470ef4f1fcb504313955cd08bf70e3be03ad098d not found: ID does not exist"
Oct 30 23:05:10.470531 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.470492    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"]
Oct 30 23:05:10.513546 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:10.513508    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-bmvsb"]
Oct 30 23:05:10.823426 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:10.823334    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:05:12.096418 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:12.096384    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" path="/var/lib/kubelet/pods/c4483810-5f34-443d-937c-9e7354fa50ef/volumes"
Oct 30 23:05:12.104905 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:12.104874    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4" podUID="d18be17e-6d08-4be0-a6df-7f4f8950214e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:15.902744 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:15.902705    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"]
Oct 30 23:05:15.982272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:15.982232    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"]
Oct 30 23:05:20.069952 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.069876    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 23:05:20.109161 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.109131    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"]
Oct 30 23:05:20.112780 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.112754    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e14e00-j9wcf"]
Oct 30 23:05:20.184953 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.184914    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18be17e-6d08-4be0-a6df-7f4f8950214e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18be17e-6d08-4be0-a6df-7f4f8950214e-plugins-config-volume\") pod \"d18be17e-6d08-4be0-a6df-7f4f8950214e\" (UID: \"d18be17e-6d08-4be0-a6df-7f4f8950214e\") "
Oct 30 23:05:20.185140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.184989    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18be17e-6d08-4be0-a6df-7f4f8950214e-kube-api-access-ss2ns\" (UniqueName: \"kubernetes.io/projected/d18be17e-6d08-4be0-a6df-7f4f8950214e-kube-api-access-ss2ns\") pod \"d18be17e-6d08-4be0-a6df-7f4f8950214e\" (UID: \"d18be17e-6d08-4be0-a6df-7f4f8950214e\") "
Oct 30 23:05:20.185446 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.185419    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18be17e-6d08-4be0-a6df-7f4f8950214e-plugins-config-volume" pod "d18be17e-6d08-4be0-a6df-7f4f8950214e" (UID: "d18be17e-6d08-4be0-a6df-7f4f8950214e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:20.187297 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.187262    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18be17e-6d08-4be0-a6df-7f4f8950214e-kube-api-access-ss2ns" pod "d18be17e-6d08-4be0-a6df-7f4f8950214e" (UID: "d18be17e-6d08-4be0-a6df-7f4f8950214e"). InnerVolumeSpecName "kube-api-access-ss2ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:20.286893 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.286855    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18be17e-6d08-4be0-a6df-7f4f8950214e-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:20.286893 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.286888    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss2ns\" (UniqueName: \"kubernetes.io/projected/d18be17e-6d08-4be0-a6df-7f4f8950214e-kube-api-access-ss2ns\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:20.487566 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.487523    4581 generic.go:356] "Generic (PLEG): container finished" podID="d18be17e-6d08-4be0-a6df-7f4f8950214e" containerID="58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196" exitCode=0
Oct 30 23:05:20.487566 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.487569    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"
Oct 30 23:05:20.487873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.487569    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4" event={"ID":"d18be17e-6d08-4be0-a6df-7f4f8950214e","Type":"ContainerDied","Data":"58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196"}
Oct 30 23:05:20.487873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.487675    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4" event={"ID":"d18be17e-6d08-4be0-a6df-7f4f8950214e","Type":"ContainerDied","Data":"b425c9bc7acfc09c5b25c770abcb1320dfe3ba7c09a235bc99eff1d2add50201"}
Oct 30 23:05:20.487873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.487717    4581 scope.go:122] "RemoveContainer" containerID="58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196"
Oct 30 23:05:20.500333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.500305    4581 scope.go:122] "RemoveContainer" containerID="58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196"
Oct 30 23:05:20.500651 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:20.500628    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196\": container with ID starting with 58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196 not found: ID does not exist" containerID="58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196"
Oct 30 23:05:20.500804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.500657    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196"} err="failed to get container status \"58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196\": rpc error: code = NotFound desc = could not find container \"58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196\": container with ID starting with 58d5c4f530592c7d92bc19c2c080724562840736b6d6bf57cf78e9698fd6a196 not found: ID does not exist"
Oct 30 23:05:20.537624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.537583    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"]
Oct 30 23:05:20.542312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.542281    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cdc122-5b697dd967-7qll4"]
Oct 30 23:05:20.903507 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.903465    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb" podUID="f1361806-d6d4-448e-94cf-8742c795b818" containerName="payload-processing" containerID="cri-o://4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80" gracePeriod=25
Oct 30 23:05:20.983625 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:20.983573    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc" podUID="aac61f39-1903-445c-ad4a-ffa29553e058" containerName="payload-pre-processing" containerID="cri-o://7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475" gracePeriod=25
Oct 30 23:05:22.095255 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:22.095199    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1468a7fd-3f9b-4899-88df-4a83749041b0" path="/var/lib/kubelet/pods/1468a7fd-3f9b-4899-88df-4a83749041b0/volumes"
Oct 30 23:05:22.095646 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:22.095618    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18be17e-6d08-4be0-a6df-7f4f8950214e" path="/var/lib/kubelet/pods/d18be17e-6d08-4be0-a6df-7f4f8950214e/volumes"
Oct 30 23:05:23.227724 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:23.227686    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb" podUID="f1361806-d6d4-448e-94cf-8742c795b818" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:23.228153 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:23.227686    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc" podUID="aac61f39-1903-445c-ad4a-ffa29553e058" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:30.324040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.323998    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"]
Oct 30 23:05:30.324600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324470    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy"
Oct 30 23:05:30.324600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324482    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy"
Oct 30 23:05:30.324600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324495    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1468a7fd-3f9b-4899-88df-4a83749041b0" containerName="revoke-keys"
Oct 30 23:05:30.324600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324500    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1468a7fd-3f9b-4899-88df-4a83749041b0" containerName="revoke-keys"
Oct 30 23:05:30.324600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324508    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d18be17e-6d08-4be0-a6df-7f4f8950214e" containerName="payload-pre-processing"
Oct 30 23:05:30.324600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324512    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18be17e-6d08-4be0-a6df-7f4f8950214e" containerName="payload-pre-processing"
Oct 30 23:05:30.324943 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324608    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4483810-5f34-443d-937c-9e7354fa50ef" containerName="istio-proxy"
Oct 30 23:05:30.324943 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324618    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1468a7fd-3f9b-4899-88df-4a83749041b0" containerName="revoke-keys"
Oct 30 23:05:30.324943 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.324624    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18be17e-6d08-4be0-a6df-7f4f8950214e" containerName="payload-pre-processing"
Oct 30 23:05:30.327910 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.327888    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.353430 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.353403    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:05:30.353620 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.353501    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:05:30.353689 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.353620    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:05:30.353744 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.353726    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:05:30.360287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.360253    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"]
Oct 30 23:05:30.490243 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.490197    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.490430 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.490395    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.490503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.490488    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqf8k\" (UniqueName: \"kubernetes.io/projected/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-kube-api-access-gqf8k\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.591126 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.591042    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.591126 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.591115    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqf8k\" (UniqueName: \"kubernetes.io/projected/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-kube-api-access-gqf8k\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.591378 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.591173    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.591885 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.591859    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.593803 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.593778    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.608182 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.608153    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqf8k\" (UniqueName: \"kubernetes.io/projected/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-kube-api-access-gqf8k\") pod \"maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.643167 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.643137    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:30.949907 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:30.949877    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"]
Oct 30 23:05:30.952660 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:05:30.952625    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbef5c3c_c80d_4aa2_8024_ccfbdb303fc5.slice/crio-f88d8023fc4fc5cef6e27dc37f966e5f8dc545ae82fba0348da6466e44fd9b1b WatchSource:0}: Error finding container f88d8023fc4fc5cef6e27dc37f966e5f8dc545ae82fba0348da6466e44fd9b1b: Status 404 returned error can't find the container with id f88d8023fc4fc5cef6e27dc37f966e5f8dc545ae82fba0348da6466e44fd9b1b
Oct 30 23:05:31.575725 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:31.575630    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955" event={"ID":"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5","Type":"ContainerStarted","Data":"0b45c750da8a16af87952319b64473fa30440b843cf71d29fff5bfaa22d0a5ea"}
Oct 30 23:05:31.575725 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:31.575700    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955" event={"ID":"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5","Type":"ContainerStarted","Data":"f88d8023fc4fc5cef6e27dc37f966e5f8dc545ae82fba0348da6466e44fd9b1b"}
Oct 30 23:05:31.614150 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:31.614099    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955" podStartSLOduration=1.6140858809999998 podStartE2EDuration="1.614085881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:31.609837147 +0000 UTC m=+3515.648776032" watchObservedRunningTime="2026-10-30 23:05:31.614085881 +0000 UTC m=+3515.653024766"
Oct 30 23:05:32.584689 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:32.584647    4581 generic.go:356] "Generic (PLEG): container finished" podID="dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5" containerID="0b45c750da8a16af87952319b64473fa30440b843cf71d29fff5bfaa22d0a5ea" exitCode=0
Oct 30 23:05:32.585169 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:32.584730    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955" event={"ID":"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5","Type":"ContainerDied","Data":"0b45c750da8a16af87952319b64473fa30440b843cf71d29fff5bfaa22d0a5ea"}
Oct 30 23:05:33.227587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.227541    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb" podUID="f1361806-d6d4-448e-94cf-8742c795b818" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:33.227806 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.227593    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc" podUID="aac61f39-1903-445c-ad4a-ffa29553e058" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:33.735866 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.735839    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:33.925731 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.925694    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-xks-ca\") pod \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") "
Oct 30 23:05:33.925953 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.925795    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-openshift-ca\") pod \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") "
Oct 30 23:05:33.925953 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.925822    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-kube-api-access-gqf8k\" (UniqueName: \"kubernetes.io/projected/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-kube-api-access-gqf8k\") pod \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\" (UID: \"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5\") "
Oct 30 23:05:33.926189 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.926164    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-openshift-ca" pod "dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5" (UID: "dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:33.928041 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.928013    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-kube-api-access-gqf8k" pod "dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5" (UID: "dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5"). InnerVolumeSpecName "kube-api-access-gqf8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:33.928041 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:33.928006    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-xks-ca" pod "dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5" (UID: "dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:34.026541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:34.026503    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:34.026541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:34.026533    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:34.026541 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:34.026545    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqf8k\" (UniqueName: \"kubernetes.io/projected/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5-kube-api-access-gqf8k\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:34.601886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:34.601856    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"
Oct 30 23:05:34.602083 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:34.601855    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955" event={"ID":"dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5","Type":"ContainerDied","Data":"f88d8023fc4fc5cef6e27dc37f966e5f8dc545ae82fba0348da6466e44fd9b1b"}
Oct 30 23:05:34.602083 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:34.601994    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88d8023fc4fc5cef6e27dc37f966e5f8dc545ae82fba0348da6466e44fd9b1b"
Oct 30 23:05:35.061315 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:35.061277    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"]
Oct 30 23:05:35.075610 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:35.075578    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"]
Oct 30 23:05:35.086478 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:35.086449    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"]
Oct 30 23:05:40.762268 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:40.762234    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"]
Oct 30 23:05:40.817642 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:40.817605    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:05:41.244180 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.244155    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 23:05:41.266665 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.266640    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 23:05:41.292253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.292200    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac61f39-1903-445c-ad4a-ffa29553e058-kube-api-access-cs7hd\" (UniqueName: \"kubernetes.io/projected/aac61f39-1903-445c-ad4a-ffa29553e058-kube-api-access-cs7hd\") pod \"aac61f39-1903-445c-ad4a-ffa29553e058\" (UID: \"aac61f39-1903-445c-ad4a-ffa29553e058\") "
Oct 30 23:05:41.292436 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.292269    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1361806-d6d4-448e-94cf-8742c795b818-kube-api-access-wdb4p\" (UniqueName: \"kubernetes.io/projected/f1361806-d6d4-448e-94cf-8742c795b818-kube-api-access-wdb4p\") pod \"f1361806-d6d4-448e-94cf-8742c795b818\" (UID: \"f1361806-d6d4-448e-94cf-8742c795b818\") "
Oct 30 23:05:41.292436 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.292337    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1361806-d6d4-448e-94cf-8742c795b818-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1361806-d6d4-448e-94cf-8742c795b818-plugins-config-volume\") pod \"f1361806-d6d4-448e-94cf-8742c795b818\" (UID: \"f1361806-d6d4-448e-94cf-8742c795b818\") "
Oct 30 23:05:41.292575 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.292455    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac61f39-1903-445c-ad4a-ffa29553e058-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac61f39-1903-445c-ad4a-ffa29553e058-plugins-config-volume\") pod \"aac61f39-1903-445c-ad4a-ffa29553e058\" (UID: \"aac61f39-1903-445c-ad4a-ffa29553e058\") "
Oct 30 23:05:41.292791 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.292762    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1361806-d6d4-448e-94cf-8742c795b818-plugins-config-volume" pod "f1361806-d6d4-448e-94cf-8742c795b818" (UID: "f1361806-d6d4-448e-94cf-8742c795b818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:41.292918 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.292806    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac61f39-1903-445c-ad4a-ffa29553e058-plugins-config-volume" pod "aac61f39-1903-445c-ad4a-ffa29553e058" (UID: "aac61f39-1903-445c-ad4a-ffa29553e058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:41.292967 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.292917    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1361806-d6d4-448e-94cf-8742c795b818-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:41.293028 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.292974    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac61f39-1903-445c-ad4a-ffa29553e058-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:41.294563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.294540    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1361806-d6d4-448e-94cf-8742c795b818-kube-api-access-wdb4p" pod "f1361806-d6d4-448e-94cf-8742c795b818" (UID: "f1361806-d6d4-448e-94cf-8742c795b818"). InnerVolumeSpecName "kube-api-access-wdb4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:41.294661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.294594    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac61f39-1903-445c-ad4a-ffa29553e058-kube-api-access-cs7hd" pod "aac61f39-1903-445c-ad4a-ffa29553e058" (UID: "aac61f39-1903-445c-ad4a-ffa29553e058"). InnerVolumeSpecName "kube-api-access-cs7hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:41.394085 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.394048    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7hd\" (UniqueName: \"kubernetes.io/projected/aac61f39-1903-445c-ad4a-ffa29553e058-kube-api-access-cs7hd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:41.394085 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.394078    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdb4p\" (UniqueName: \"kubernetes.io/projected/f1361806-d6d4-448e-94cf-8742c795b818-kube-api-access-wdb4p\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:41.656841 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.656747    4581 generic.go:356] "Generic (PLEG): container finished" podID="aac61f39-1903-445c-ad4a-ffa29553e058" containerID="7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475" exitCode=0
Oct 30 23:05:41.656841 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.656788    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"
Oct 30 23:05:41.657068 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.656835    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc" event={"ID":"aac61f39-1903-445c-ad4a-ffa29553e058","Type":"ContainerDied","Data":"7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475"}
Oct 30 23:05:41.657068 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.656887    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc" event={"ID":"aac61f39-1903-445c-ad4a-ffa29553e058","Type":"ContainerDied","Data":"e14cbc2b33041c56682d8827ed722ba89e23ed84faa640ff018c174642a5a87a"}
Oct 30 23:05:41.657068 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.656914    4581 scope.go:122] "RemoveContainer" containerID="7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475"
Oct 30 23:05:41.658682 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.658655    4581 generic.go:356] "Generic (PLEG): container finished" podID="f1361806-d6d4-448e-94cf-8742c795b818" containerID="4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80" exitCode=0
Oct 30 23:05:41.658799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.658766    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb" event={"ID":"f1361806-d6d4-448e-94cf-8742c795b818","Type":"ContainerDied","Data":"4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80"}
Oct 30 23:05:41.658799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.658797    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb" event={"ID":"f1361806-d6d4-448e-94cf-8742c795b818","Type":"ContainerDied","Data":"2a7095e339582f39488809e54af89d7e2838cab3e8c7336c0a91e1e2006efe9a"}
Oct 30 23:05:41.658889 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.658844    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"
Oct 30 23:05:41.674273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.674253    4581 scope.go:122] "RemoveContainer" containerID="7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475"
Oct 30 23:05:41.674663 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:41.674640    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475\": container with ID starting with 7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475 not found: ID does not exist" containerID="7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475"
Oct 30 23:05:41.674753 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.674675    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475"} err="failed to get container status \"7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475\": rpc error: code = NotFound desc = could not find container \"7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475\": container with ID starting with 7609d30fbbf405168910f942c932ab5b3e63353ac72690a190cf44744c11d475 not found: ID does not exist"
Oct 30 23:05:41.674753 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.674701    4581 scope.go:122] "RemoveContainer" containerID="4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80"
Oct 30 23:05:41.684821 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.684797    4581 scope.go:122] "RemoveContainer" containerID="4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80"
Oct 30 23:05:41.685132 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:41.685107    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80\": container with ID starting with 4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80 not found: ID does not exist" containerID="4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80"
Oct 30 23:05:41.685192 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.685143    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80"} err="failed to get container status \"4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80\": rpc error: code = NotFound desc = could not find container \"4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80\": container with ID starting with 4fbdec31c45f6534142944a97d75c1277b530aa18797fcd1a4e9e9881bea7d80 not found: ID does not exist"
Oct 30 23:05:41.709326 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.709300    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"]
Oct 30 23:05:41.713970 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.713948    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e14e00-bdf56dd9f-67qgc"]
Oct 30 23:05:41.743112 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.743076    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"]
Oct 30 23:05:41.746564 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:41.746534    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e14e00-56ff64bff5-ht6tb"]
Oct 30 23:05:42.095736 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.095698    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac61f39-1903-445c-ad4a-ffa29553e058" path="/var/lib/kubelet/pods/aac61f39-1903-445c-ad4a-ffa29553e058/volumes"
Oct 30 23:05:42.096110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.096077    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1361806-d6d4-448e-94cf-8742c795b818" path="/var/lib/kubelet/pods/f1361806-d6d4-448e-94cf-8742c795b818/volumes"
Oct 30 23:05:42.378276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378237    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"]
Oct 30 23:05:42.378710 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378691    4581 cpu_manager.go:414] "containerMap: removing container" podUID="f1361806-d6d4-448e-94cf-8742c795b818" containerName="payload-processing"
Oct 30 23:05:42.378815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378712    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1361806-d6d4-448e-94cf-8742c795b818" containerName="payload-processing"
Oct 30 23:05:42.378815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378729    4581 cpu_manager.go:414] "containerMap: removing container" podUID="aac61f39-1903-445c-ad4a-ffa29553e058" containerName="payload-pre-processing"
Oct 30 23:05:42.378815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378736    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac61f39-1903-445c-ad4a-ffa29553e058" containerName="payload-pre-processing"
Oct 30 23:05:42.378815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378787    4581 cpu_manager.go:414] "containerMap: removing container" podUID="dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5" containerName="revoke-keys"
Oct 30 23:05:42.378815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378795    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5" containerName="revoke-keys"
Oct 30 23:05:42.379085 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378908    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5" containerName="revoke-keys"
Oct 30 23:05:42.379085 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378926    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1361806-d6d4-448e-94cf-8742c795b818" containerName="payload-processing"
Oct 30 23:05:42.379085 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.378935    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac61f39-1903-445c-ad4a-ffa29553e058" containerName="payload-pre-processing"
Oct 30 23:05:42.383736 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.383713    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.396592 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.396568    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:05:42.404054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.404024    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.404150 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.404094    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.404212 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.404151    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dpn\" (UniqueName: \"kubernetes.io/projected/d8bbd476-8aef-4abe-924c-d6e4547612fe-kube-api-access-q8dpn\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.406416 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.406392    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:05:42.406999 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.406982    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:05:42.406999 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.406996    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:05:42.410823 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.410803    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"]
Oct 30 23:05:42.505563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.505514    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.505788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.505610    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.505788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.505640    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dpn\" (UniqueName: \"kubernetes.io/projected/d8bbd476-8aef-4abe-924c-d6e4547612fe-kube-api-access-q8dpn\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.506434 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.506358    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.508173 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.508148    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.522881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.522859    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dpn\" (UniqueName: \"kubernetes.io/projected/d8bbd476-8aef-4abe-924c-d6e4547612fe-kube-api-access-q8dpn\") pod \"maas-api-revoke-sub-d0b7132f-shjq8\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:42.697106 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:42.697018    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:43.003684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:43.003653    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"]
Oct 30 23:05:43.006700 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:05:43.006662    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bbd476_8aef_4abe_924c_d6e4547612fe.slice/crio-e0e0c5ecf6727361c3e50401d1af3d0a493e39bc55e7cda1764319e210e81074 WatchSource:0}: Error finding container e0e0c5ecf6727361c3e50401d1af3d0a493e39bc55e7cda1764319e210e81074: Status 404 returned error can't find the container with id e0e0c5ecf6727361c3e50401d1af3d0a493e39bc55e7cda1764319e210e81074
Oct 30 23:05:43.677657 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:43.677620    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8" event={"ID":"d8bbd476-8aef-4abe-924c-d6e4547612fe","Type":"ContainerStarted","Data":"aa912c3eaf0e0608d1f74aa734b684c754efa6f058126db8abf8f28ea1808d48"}
Oct 30 23:05:43.678095 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:43.677687    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8" event={"ID":"d8bbd476-8aef-4abe-924c-d6e4547612fe","Type":"ContainerStarted","Data":"e0e0c5ecf6727361c3e50401d1af3d0a493e39bc55e7cda1764319e210e81074"}
Oct 30 23:05:43.715509 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:43.715456    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8" podStartSLOduration=1.71544202 podStartE2EDuration="1.71544202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:43.712523131 +0000 UTC m=+3527.751462017" watchObservedRunningTime="2026-10-30 23:05:43.71544202 +0000 UTC m=+3527.754380905"
Oct 30 23:05:44.687932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:44.687900    4581 generic.go:356] "Generic (PLEG): container finished" podID="d8bbd476-8aef-4abe-924c-d6e4547612fe" containerID="aa912c3eaf0e0608d1f74aa734b684c754efa6f058126db8abf8f28ea1808d48" exitCode=0
Oct 30 23:05:44.688378 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:44.687989    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8" event={"ID":"d8bbd476-8aef-4abe-924c-d6e4547612fe","Type":"ContainerDied","Data":"aa912c3eaf0e0608d1f74aa734b684c754efa6f058126db8abf8f28ea1808d48"}
Oct 30 23:05:45.763798 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:45.763534    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8" podUID="14657837-22d6-40d8-af6f-a6126c9ae497" containerName="payload-processing" containerID="cri-o://07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433" gracePeriod=25
Oct 30 23:05:45.856660 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:45.856627    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:45.941026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:45.940990    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-openshift-ca\") pod \"d8bbd476-8aef-4abe-924c-d6e4547612fe\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") "
Oct 30 23:05:45.941207 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:45.941063    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bbd476-8aef-4abe-924c-d6e4547612fe-kube-api-access-q8dpn\" (UniqueName: \"kubernetes.io/projected/d8bbd476-8aef-4abe-924c-d6e4547612fe-kube-api-access-q8dpn\") pod \"d8bbd476-8aef-4abe-924c-d6e4547612fe\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") "
Oct 30 23:05:45.941207 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:45.941114    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-xks-ca\") pod \"d8bbd476-8aef-4abe-924c-d6e4547612fe\" (UID: \"d8bbd476-8aef-4abe-924c-d6e4547612fe\") "
Oct 30 23:05:45.941454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:45.941428    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-openshift-ca" pod "d8bbd476-8aef-4abe-924c-d6e4547612fe" (UID: "d8bbd476-8aef-4abe-924c-d6e4547612fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:45.943469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:45.943444    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bbd476-8aef-4abe-924c-d6e4547612fe-kube-api-access-q8dpn" pod "d8bbd476-8aef-4abe-924c-d6e4547612fe" (UID: "d8bbd476-8aef-4abe-924c-d6e4547612fe"). InnerVolumeSpecName "kube-api-access-q8dpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:45.943598 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:45.943496    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-xks-ca" pod "d8bbd476-8aef-4abe-924c-d6e4547612fe" (UID: "d8bbd476-8aef-4abe-924c-d6e4547612fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:46.042412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:46.042370    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8dpn\" (UniqueName: \"kubernetes.io/projected/d8bbd476-8aef-4abe-924c-d6e4547612fe-kube-api-access-q8dpn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:46.042412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:46.042399    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:46.042412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:46.042410    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bbd476-8aef-4abe-924c-d6e4547612fe-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:46.707967 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:46.707932    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"
Oct 30 23:05:46.707967 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:46.707956    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8" event={"ID":"d8bbd476-8aef-4abe-924c-d6e4547612fe","Type":"ContainerDied","Data":"e0e0c5ecf6727361c3e50401d1af3d0a493e39bc55e7cda1764319e210e81074"}
Oct 30 23:05:46.708191 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:46.708011    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e0c5ecf6727361c3e50401d1af3d0a493e39bc55e7cda1764319e210e81074"
Oct 30 23:05:47.554281 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:47.554244    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8" podUID="14657837-22d6-40d8-af6f-a6126c9ae497" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:50.082451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.082407    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main" containerID="cri-o://7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb" gracePeriod=45
Oct 30 23:05:50.096330 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.096279    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main" containerID="cri-o://313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8" gracePeriod=45
Oct 30 23:05:50.107179 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.107134    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main" containerID="cri-o://b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889" gracePeriod=45
Oct 30 23:05:50.377231 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.377185    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 23:05:50.404084 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.404045    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 23:05:50.407586 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.407564    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 23:05:50.484703 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484674    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-dshm\") pod \"337f84f2-d4e6-468a-9341-16b93e1b785b\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") "
Oct 30 23:05:50.484891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484715    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5cc362-a99e-4622-9e94-a263b53ded58-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5cc362-a99e-4622-9e94-a263b53ded58-tls-certs\") pod \"4a5cc362-a99e-4622-9e94-a263b53ded58\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") "
Oct 30 23:05:50.484891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484735    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5cc362-a99e-4622-9e94-a263b53ded58-kube-api-access-mvjrn\" (UniqueName: \"kubernetes.io/projected/4a5cc362-a99e-4622-9e94-a263b53ded58-kube-api-access-mvjrn\") pod \"4a5cc362-a99e-4622-9e94-a263b53ded58\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") "
Oct 30 23:05:50.484891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484759    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-home\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-home\") pod \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") "
Oct 30 23:05:50.484891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484783    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-dshm\") pod \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") "
Oct 30 23:05:50.484891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484808    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-model-cache\") pod \"337f84f2-d4e6-468a-9341-16b93e1b785b\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") "
Oct 30 23:05:50.484891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484875    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-dshm\") pod \"4a5cc362-a99e-4622-9e94-a263b53ded58\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") "
Oct 30 23:05:50.485278 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484908    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-kube-api-access-pd9zp\" (UniqueName: \"kubernetes.io/projected/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-kube-api-access-pd9zp\") pod \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") "
Oct 30 23:05:50.485278 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484949    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-home\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-home\") pod \"337f84f2-d4e6-468a-9341-16b93e1b785b\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") "
Oct 30 23:05:50.485278 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.484988    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-home\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-home\") pod \"4a5cc362-a99e-4622-9e94-a263b53ded58\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") "
Oct 30 23:05:50.485278 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485037    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-tmp-dir\") pod \"337f84f2-d4e6-468a-9341-16b93e1b785b\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") "
Oct 30 23:05:50.485278 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485065    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-model-cache\") pod \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") "
Oct 30 23:05:50.485278 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485120    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-model-cache\") pod \"4a5cc362-a99e-4622-9e94-a263b53ded58\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") "
Oct 30 23:05:50.485278 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485190    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tls-certs\" (UniqueName: \"kubernetes.io/secret/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tls-certs\") pod \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") "
Oct 30 23:05:50.485278 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485238    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337f84f2-d4e6-468a-9341-16b93e1b785b-kube-api-access-qfl5k\" (UniqueName: \"kubernetes.io/projected/337f84f2-d4e6-468a-9341-16b93e1b785b-kube-api-access-qfl5k\") pod \"337f84f2-d4e6-468a-9341-16b93e1b785b\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") "
Oct 30 23:05:50.485975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485278    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-model-cache" pod "337f84f2-d4e6-468a-9341-16b93e1b785b" (UID: "337f84f2-d4e6-468a-9341-16b93e1b785b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.485975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485291    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-tmp-dir\") pod \"4a5cc362-a99e-4622-9e94-a263b53ded58\" (UID: \"4a5cc362-a99e-4622-9e94-a263b53ded58\") "
Oct 30 23:05:50.485975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485355    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tmp-dir\") pod \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\" (UID: \"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08\") "
Oct 30 23:05:50.485975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485510    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-home" pod "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" (UID: "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.485975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485607    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-tmp-dir" pod "4a5cc362-a99e-4622-9e94-a263b53ded58" (UID: "4a5cc362-a99e-4622-9e94-a263b53ded58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.485975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485637    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-home" pod "4a5cc362-a99e-4622-9e94-a263b53ded58" (UID: "4a5cc362-a99e-4622-9e94-a263b53ded58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.485975 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.485832    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-model-cache" pod "4a5cc362-a99e-4622-9e94-a263b53ded58" (UID: "4a5cc362-a99e-4622-9e94-a263b53ded58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.486391 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.486014    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-tmp-dir" pod "337f84f2-d4e6-468a-9341-16b93e1b785b" (UID: "337f84f2-d4e6-468a-9341-16b93e1b785b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.486391 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.486192    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-model-cache" pod "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" (UID: "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.486391 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.486276    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337f84f2-d4e6-468a-9341-16b93e1b785b-tls-certs\" (UniqueName: \"kubernetes.io/secret/337f84f2-d4e6-468a-9341-16b93e1b785b-tls-certs\") pod \"337f84f2-d4e6-468a-9341-16b93e1b785b\" (UID: \"337f84f2-d4e6-468a-9341-16b93e1b785b\") "
Oct 30 23:05:50.486712 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.486687    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tmp-dir" pod "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" (UID: "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.486862 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.486841    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.487152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.486858    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-home" pod "337f84f2-d4e6-468a-9341-16b93e1b785b" (UID: "337f84f2-d4e6-468a-9341-16b93e1b785b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.487152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.486870    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.487152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.486904    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.489536 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.489490    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-kube-api-access-pd9zp" pod "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" (UID: "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08"). InnerVolumeSpecName "kube-api-access-pd9zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:50.490388 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.490313    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337f84f2-d4e6-468a-9341-16b93e1b785b-tls-certs" pod "337f84f2-d4e6-468a-9341-16b93e1b785b" (UID: "337f84f2-d4e6-468a-9341-16b93e1b785b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:50.491628 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.491600    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tls-certs" pod "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" (UID: "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:50.491906 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.491886    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337f84f2-d4e6-468a-9341-16b93e1b785b-kube-api-access-qfl5k" pod "337f84f2-d4e6-468a-9341-16b93e1b785b" (UID: "337f84f2-d4e6-468a-9341-16b93e1b785b"). InnerVolumeSpecName "kube-api-access-qfl5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:50.492799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.492732    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5cc362-a99e-4622-9e94-a263b53ded58-tls-certs" pod "4a5cc362-a99e-4622-9e94-a263b53ded58" (UID: "4a5cc362-a99e-4622-9e94-a263b53ded58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:50.492799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.492753    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5cc362-a99e-4622-9e94-a263b53ded58-kube-api-access-mvjrn" pod "4a5cc362-a99e-4622-9e94-a263b53ded58" (UID: "4a5cc362-a99e-4622-9e94-a263b53ded58"). InnerVolumeSpecName "kube-api-access-mvjrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:50.492799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.492794    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-dshm" pod "337f84f2-d4e6-468a-9341-16b93e1b785b" (UID: "337f84f2-d4e6-468a-9341-16b93e1b785b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.493248 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.493189    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-dshm" pod "4a5cc362-a99e-4622-9e94-a263b53ded58" (UID: "4a5cc362-a99e-4622-9e94-a263b53ded58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.494131 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.494105    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-dshm" pod "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" (UID: "7fef4a56-7d62-4fc7-b3f8-1d372b71fe08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.588310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588273    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588305    4581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-home\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588314    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588326    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588334    4581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-model-cache\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588342    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588351    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfl5k\" (UniqueName: \"kubernetes.io/projected/337f84f2-d4e6-468a-9341-16b93e1b785b-kube-api-access-qfl5k\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588361    4581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-tmp-dir\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588369    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/337f84f2-d4e6-468a-9341-16b93e1b785b-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588376    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/337f84f2-d4e6-468a-9341-16b93e1b785b-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588383    4581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5cc362-a99e-4622-9e94-a263b53ded58-tls-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588390    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjrn\" (UniqueName: \"kubernetes.io/projected/4a5cc362-a99e-4622-9e94-a263b53ded58-kube-api-access-mvjrn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588399    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588406    4581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5cc362-a99e-4622-9e94-a263b53ded58-dshm\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.588558 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.588414    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd9zp\" (UniqueName: \"kubernetes.io/projected/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08-kube-api-access-pd9zp\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.749040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.748940    4581 generic.go:356] "Generic (PLEG): container finished" podID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerID="b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889" exitCode=0
Oct 30 23:05:50.749040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.748985    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"
Oct 30 23:05:50.749040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.749017    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" event={"ID":"4a5cc362-a99e-4622-9e94-a263b53ded58","Type":"ContainerDied","Data":"b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889"}
Oct 30 23:05:50.749462 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.749060    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz" event={"ID":"4a5cc362-a99e-4622-9e94-a263b53ded58","Type":"ContainerDied","Data":"d3f1593e48dc279e4b19f767cade55f72303a8b50899ff3b97a858b3fa6f2dbc"}
Oct 30 23:05:50.749462 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.749078    4581 scope.go:122] "RemoveContainer" containerID="b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889"
Oct 30 23:05:50.750546 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.750519    4581 generic.go:356] "Generic (PLEG): container finished" podID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerID="7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb" exitCode=0
Oct 30 23:05:50.750644 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.750553    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"
Oct 30 23:05:50.750644 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.750612    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" event={"ID":"337f84f2-d4e6-468a-9341-16b93e1b785b","Type":"ContainerDied","Data":"7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb"}
Oct 30 23:05:50.750808 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.750722    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c" event={"ID":"337f84f2-d4e6-468a-9341-16b93e1b785b","Type":"ContainerDied","Data":"2e80af4bfa60288db300d33918fdc57921cdb6d9d30acf9e87e72f0a881628f0"}
Oct 30 23:05:50.752355 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.752331    4581 generic.go:356] "Generic (PLEG): container finished" podID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerID="313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8" exitCode=0
Oct 30 23:05:50.752480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.752355    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" event={"ID":"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08","Type":"ContainerDied","Data":"313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8"}
Oct 30 23:05:50.752480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.752379    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"
Oct 30 23:05:50.752480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.752394    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg" event={"ID":"7fef4a56-7d62-4fc7-b3f8-1d372b71fe08","Type":"ContainerDied","Data":"b6be8b12a3a2b21d1b96c24f4d684e0a04b329c71f53e8b38fbf047b3bbc9c2c"}
Oct 30 23:05:50.761955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.761933    4581 scope.go:122] "RemoveContainer" containerID="b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889"
Oct 30 23:05:50.762306 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:50.762285    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889\": container with ID starting with b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889 not found: ID does not exist" containerID="b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889"
Oct 30 23:05:50.762400 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.762322    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889"} err="failed to get container status \"b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889\": rpc error: code = NotFound desc = could not find container \"b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889\": container with ID starting with b4d1fa03440e6837cb8967735ad02627166bedd3d3958df6c21e2ed8bd66b889 not found: ID does not exist"
Oct 30 23:05:50.762400 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.762345    4581 scope.go:122] "RemoveContainer" containerID="7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb"
Oct 30 23:05:50.771838 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.771820    4581 scope.go:122] "RemoveContainer" containerID="7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb"
Oct 30 23:05:50.772147 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:50.772129    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb\": container with ID starting with 7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb not found: ID does not exist" containerID="7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb"
Oct 30 23:05:50.772246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.772158    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb"} err="failed to get container status \"7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb\": rpc error: code = NotFound desc = could not find container \"7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb\": container with ID starting with 7c9800d55b42999b3020e17c249489f16ec623c11d69788eb3a9ed5f4b441bdb not found: ID does not exist"
Oct 30 23:05:50.772246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.772182    4581 scope.go:122] "RemoveContainer" containerID="313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8"
Oct 30 23:05:50.781606 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.781584    4581 scope.go:122] "RemoveContainer" containerID="313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8"
Oct 30 23:05:50.781938 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:05:50.781918    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8\": container with ID starting with 313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8 not found: ID does not exist" containerID="313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8"
Oct 30 23:05:50.781980 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.781947    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8"} err="failed to get container status \"313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8\": rpc error: code = NotFound desc = could not find container \"313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8\": container with ID starting with 313f9cd55ea2ff3a0b74734a7fbf44d3b1d94eaea81972b66db5d6d4e385a8f8 not found: ID does not exist"
Oct 30 23:05:50.819743 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.819710    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"]
Oct 30 23:05:50.829613 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.829588    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/premium-w3-d52c59-kserve-556b5fc797-mkncg"]
Oct 30 23:05:50.859607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.859576    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"]
Oct 30 23:05:50.862892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.862865    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-2-w3-d52c59-kserve-5756975d7-ddl2c"]
Oct 30 23:05:50.893057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.893028    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"]
Oct 30 23:05:50.896046 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:50.896023    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/simulator-w3-d52c59-kserve-5ff6c89cc6-nhsnz"]
Oct 30 23:05:52.095241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:52.095181    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" path="/var/lib/kubelet/pods/337f84f2-d4e6-468a-9341-16b93e1b785b/volumes"
Oct 30 23:05:52.095817 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:52.095796    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" path="/var/lib/kubelet/pods/4a5cc362-a99e-4622-9e94-a263b53ded58/volumes"
Oct 30 23:05:52.096241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:52.096204    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" path="/var/lib/kubelet/pods/7fef4a56-7d62-4fc7-b3f8-1d372b71fe08/volumes"
Oct 30 23:05:56.055607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:56.055570    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"]
Oct 30 23:05:56.065497 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:56.065468    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e62679-459p7"]
Oct 30 23:05:56.096104 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:56.096071    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa" path="/var/lib/kubelet/pods/d31ad848-8ea6-42eb-9aa8-0a8aa4f706aa/volumes"
Oct 30 23:05:57.553517 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:57.553483    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8" podUID="14657837-22d6-40d8-af6f-a6126c9ae497" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:58.189618 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:58.189580    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"]
Oct 30 23:05:58.197453 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:05:58.197424    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d52c59-z4955"]
Oct 30 23:06:00.095636 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:00.095598    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5" path="/var/lib/kubelet/pods/dbef5c3c-c80d-4aa2-8024-ccfbdb303fc5/volumes"
Oct 30 23:06:03.205513 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:03.205478    4581 scope.go:122] "RemoveContainer" containerID="4fb1a460a939ce5fcc2be392f155c6a20b22653b53e833007c2ade6a96715785"
Oct 30 23:06:06.023423 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.023392    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 23:06:06.027823 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.027755    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"]
Oct 30 23:06:06.037598 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.037568    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c443f6-9cjnc"]
Oct 30 23:06:06.096022 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.095991    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff01d2d-904d-4b97-8f67-f928208e138c" path="/var/lib/kubelet/pods/bff01d2d-904d-4b97-8f67-f928208e138c/volumes"
Oct 30 23:06:06.138190 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.138115    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14657837-22d6-40d8-af6f-a6126c9ae497-kube-api-access-q5hzz\" (UniqueName: \"kubernetes.io/projected/14657837-22d6-40d8-af6f-a6126c9ae497-kube-api-access-q5hzz\") pod \"14657837-22d6-40d8-af6f-a6126c9ae497\" (UID: \"14657837-22d6-40d8-af6f-a6126c9ae497\") "
Oct 30 23:06:06.138393 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.138195    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14657837-22d6-40d8-af6f-a6126c9ae497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14657837-22d6-40d8-af6f-a6126c9ae497-plugins-config-volume\") pod \"14657837-22d6-40d8-af6f-a6126c9ae497\" (UID: \"14657837-22d6-40d8-af6f-a6126c9ae497\") "
Oct 30 23:06:06.138640 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.138616    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14657837-22d6-40d8-af6f-a6126c9ae497-plugins-config-volume" pod "14657837-22d6-40d8-af6f-a6126c9ae497" (UID: "14657837-22d6-40d8-af6f-a6126c9ae497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:06:06.140550 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.140520    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14657837-22d6-40d8-af6f-a6126c9ae497-kube-api-access-q5hzz" pod "14657837-22d6-40d8-af6f-a6126c9ae497" (UID: "14657837-22d6-40d8-af6f-a6126c9ae497"). InnerVolumeSpecName "kube-api-access-q5hzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:06.239461 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.239420    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5hzz\" (UniqueName: \"kubernetes.io/projected/14657837-22d6-40d8-af6f-a6126c9ae497-kube-api-access-q5hzz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:06:06.239461 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.239453    4581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14657837-22d6-40d8-af6f-a6126c9ae497-plugins-config-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:06:06.874911 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.874875    4581 generic.go:356] "Generic (PLEG): container finished" podID="14657837-22d6-40d8-af6f-a6126c9ae497" containerID="07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433" exitCode=0
Oct 30 23:06:06.875128 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.874951    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"
Oct 30 23:06:06.875128 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.874971    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8" event={"ID":"14657837-22d6-40d8-af6f-a6126c9ae497","Type":"ContainerDied","Data":"07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433"}
Oct 30 23:06:06.875128 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.875040    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8" event={"ID":"14657837-22d6-40d8-af6f-a6126c9ae497","Type":"ContainerDied","Data":"cb26459bc9157d5665d4701dd068da0d2e19e877d548db4bd390c503f8313518"}
Oct 30 23:06:06.875128 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.875075    4581 scope.go:122] "RemoveContainer" containerID="07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433"
Oct 30 23:06:06.888632 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.888485    4581 scope.go:122] "RemoveContainer" containerID="07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433"
Oct 30 23:06:06.888845 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:06:06.888821    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433\": container with ID starting with 07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433 not found: ID does not exist" containerID="07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433"
Oct 30 23:06:06.888929 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.888857    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433"} err="failed to get container status \"07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433\": rpc error: code = NotFound desc = could not find container \"07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433\": container with ID starting with 07406c1e1ac874b895e51633440ca154d95460145b543b922cd7008c466c2433 not found: ID does not exist"
Oct 30 23:06:06.923552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.923519    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"]
Oct 30 23:06:06.935568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:06.935399    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d52c59-f75cff857-zdrn8"]
Oct 30 23:06:08.094542 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:08.094508    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14657837-22d6-40d8-af6f-a6126c9ae497" path="/var/lib/kubelet/pods/14657837-22d6-40d8-af6f-a6126c9ae497/volumes"
Oct 30 23:06:10.817232 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:06:10.817171    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:06:31.732077 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732028    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"]
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732441    4581 cpu_manager.go:414] "containerMap: removing container" podUID="14657837-22d6-40d8-af6f-a6126c9ae497" containerName="payload-processing"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732457    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14657837-22d6-40d8-af6f-a6126c9ae497" containerName="payload-processing"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732480    4581 cpu_manager.go:414] "containerMap: removing container" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732485    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732502    4581 cpu_manager.go:414] "containerMap: removing container" podUID="d8bbd476-8aef-4abe-924c-d6e4547612fe" containerName="revoke-keys"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732507    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bbd476-8aef-4abe-924c-d6e4547612fe" containerName="revoke-keys"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732515    4581 cpu_manager.go:414] "containerMap: removing container" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732519    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732529    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main"
Oct 30 23:06:31.732547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732533    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main"
Oct 30 23:06:31.732884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732589    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="14657837-22d6-40d8-af6f-a6126c9ae497" containerName="payload-processing"
Oct 30 23:06:31.732884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732602    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="337f84f2-d4e6-468a-9341-16b93e1b785b" containerName="main"
Oct 30 23:06:31.732884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732609    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bbd476-8aef-4abe-924c-d6e4547612fe" containerName="revoke-keys"
Oct 30 23:06:31.732884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732615    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5cc362-a99e-4622-9e94-a263b53ded58" containerName="main"
Oct 30 23:06:31.732884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.732621    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fef4a56-7d62-4fc7-b3f8-1d372b71fe08" containerName="main"
Oct 30 23:06:31.735972 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.735955    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.752675 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.752651    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:06:31.758267 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.758236    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.758405 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.758354    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvj8p\" (UniqueName: \"kubernetes.io/projected/ac4203df-9064-48dd-8e87-dbd582e681b2-kube-api-access-jvj8p\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.758405 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.758385    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.762892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.762834    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:06:31.763031 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.763000    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:06:31.763088 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.763068    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:06:31.766593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.766571    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"]
Oct 30 23:06:31.859364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.859324    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvj8p\" (UniqueName: \"kubernetes.io/projected/ac4203df-9064-48dd-8e87-dbd582e681b2-kube-api-access-jvj8p\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.859587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.859372    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.859587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.859398    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.860051 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.860022    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.861870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.861844    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:31.879042 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:31.879009    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvj8p\" (UniqueName: \"kubernetes.io/projected/ac4203df-9064-48dd-8e87-dbd582e681b2-kube-api-access-jvj8p\") pod \"maas-api-revoke-sub-85237e45-dsnvx\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:32.048888 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:32.048800    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:32.350994 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:32.350968    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"]
Oct 30 23:06:32.353710 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:06:32.353679    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4203df_9064_48dd_8e87_dbd582e681b2.slice/crio-61d6662bb7c8780155ab4b0f14d032623fa197b6b3a9fec3278be6b514f767a5 WatchSource:0}: Error finding container 61d6662bb7c8780155ab4b0f14d032623fa197b6b3a9fec3278be6b514f767a5: Status 404 returned error can't find the container with id 61d6662bb7c8780155ab4b0f14d032623fa197b6b3a9fec3278be6b514f767a5
Oct 30 23:06:33.062488 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:33.062393    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx" event={"ID":"ac4203df-9064-48dd-8e87-dbd582e681b2","Type":"ContainerStarted","Data":"a868dd77781b973aa7d458eb8e387b1ef85a73e90ba078de40c95a6af3a7d49e"}
Oct 30 23:06:33.062488 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:33.062455    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx" event={"ID":"ac4203df-9064-48dd-8e87-dbd582e681b2","Type":"ContainerStarted","Data":"61d6662bb7c8780155ab4b0f14d032623fa197b6b3a9fec3278be6b514f767a5"}
Oct 30 23:06:33.098280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:33.098231    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx" podStartSLOduration=2.098200882 podStartE2EDuration="2.098200882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:33.096502455 +0000 UTC m=+3577.135441341" watchObservedRunningTime="2026-10-30 23:06:33.098200882 +0000 UTC m=+3577.137139758"
Oct 30 23:06:34.071249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:34.071181    4581 generic.go:356] "Generic (PLEG): container finished" podID="ac4203df-9064-48dd-8e87-dbd582e681b2" containerID="a868dd77781b973aa7d458eb8e387b1ef85a73e90ba078de40c95a6af3a7d49e" exitCode=0
Oct 30 23:06:34.071725 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:34.071261    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx" event={"ID":"ac4203df-9064-48dd-8e87-dbd582e681b2","Type":"ContainerDied","Data":"a868dd77781b973aa7d458eb8e387b1ef85a73e90ba078de40c95a6af3a7d49e"}
Oct 30 23:06:35.222510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.222483    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:35.290525 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.290489    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-openshift-ca\") pod \"ac4203df-9064-48dd-8e87-dbd582e681b2\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") "
Oct 30 23:06:35.290733 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.290615    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-xks-ca\") pod \"ac4203df-9064-48dd-8e87-dbd582e681b2\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") "
Oct 30 23:06:35.290733 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.290668    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4203df-9064-48dd-8e87-dbd582e681b2-kube-api-access-jvj8p\" (UniqueName: \"kubernetes.io/projected/ac4203df-9064-48dd-8e87-dbd582e681b2-kube-api-access-jvj8p\") pod \"ac4203df-9064-48dd-8e87-dbd582e681b2\" (UID: \"ac4203df-9064-48dd-8e87-dbd582e681b2\") "
Oct 30 23:06:35.290955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.290914    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-openshift-ca" pod "ac4203df-9064-48dd-8e87-dbd582e681b2" (UID: "ac4203df-9064-48dd-8e87-dbd582e681b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:06:35.293072 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.293037    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-xks-ca" pod "ac4203df-9064-48dd-8e87-dbd582e681b2" (UID: "ac4203df-9064-48dd-8e87-dbd582e681b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:35.293188 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.293102    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4203df-9064-48dd-8e87-dbd582e681b2-kube-api-access-jvj8p" pod "ac4203df-9064-48dd-8e87-dbd582e681b2" (UID: "ac4203df-9064-48dd-8e87-dbd582e681b2"). InnerVolumeSpecName "kube-api-access-jvj8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:35.391404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.391372    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:06:35.391404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.391402    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvj8p\" (UniqueName: \"kubernetes.io/projected/ac4203df-9064-48dd-8e87-dbd582e681b2-kube-api-access-jvj8p\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:06:35.391633 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:35.391412    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4203df-9064-48dd-8e87-dbd582e681b2-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:06:36.093512 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:36.093483    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"
Oct 30 23:06:36.095561 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:36.095529    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx" event={"ID":"ac4203df-9064-48dd-8e87-dbd582e681b2","Type":"ContainerDied","Data":"61d6662bb7c8780155ab4b0f14d032623fa197b6b3a9fec3278be6b514f767a5"}
Oct 30 23:06:36.095720 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:36.095586    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61d6662bb7c8780155ab4b0f14d032623fa197b6b3a9fec3278be6b514f767a5"
Oct 30 23:06:40.817534 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:06:40.817494    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:06:44.233276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.233207    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"]
Oct 30 23:06:44.233945 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.233926    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ac4203df-9064-48dd-8e87-dbd582e681b2" containerName="revoke-keys"
Oct 30 23:06:44.234045 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.233947    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4203df-9064-48dd-8e87-dbd582e681b2" containerName="revoke-keys"
Oct 30 23:06:44.234104 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.234056    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4203df-9064-48dd-8e87-dbd582e681b2" containerName="revoke-keys"
Oct 30 23:06:44.237387 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.237364    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.261452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.261423    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"]
Oct 30 23:06:44.274504 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274448    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.274504 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274495    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.274774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274521    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.274774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274558    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.274774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274629    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjds6\" (UniqueName: \"kubernetes.io/projected/79c1511e-3651-415c-a1f7-96b1d89b447a-kube-api-access-bjds6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.274774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274655    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.274774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274702    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.274774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274732    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c1511e-3651-415c-a1f7-96b1d89b447a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.275069 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.274812    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.375981    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376017    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376037    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376060    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376101    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjds6\" (UniqueName: \"kubernetes.io/projected/79c1511e-3651-415c-a1f7-96b1d89b447a-kube-api-access-bjds6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376117    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376145    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376163    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c1511e-3651-415c-a1f7-96b1d89b447a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376333 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376255    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376727 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376499    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376730    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.376794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376754    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.377081 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.376811    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.377171 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.377151    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c1511e-3651-415c-a1f7-96b1d89b447a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.378632 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.378605    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.378959 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.378935    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.393825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.393799    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c1511e-3651-415c-a1f7-96b1d89b447a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.393929 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.393831    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjds6\" (UniqueName: \"kubernetes.io/projected/79c1511e-3651-415c-a1f7-96b1d89b447a-kube-api-access-bjds6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hcq62\" (UID: \"79c1511e-3651-415c-a1f7-96b1d89b447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.551299 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.551180    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:44.877522 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.877489    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"]
Oct 30 23:06:44.880679 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:06:44.880653    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c1511e_3651_415c_a1f7_96b1d89b447a.slice/crio-b75e49fb4485599efd541985979548c955eba69820ff9b7d36373224b4dc09a4 WatchSource:0}: Error finding container b75e49fb4485599efd541985979548c955eba69820ff9b7d36373224b4dc09a4: Status 404 returned error can't find the container with id b75e49fb4485599efd541985979548c955eba69820ff9b7d36373224b4dc09a4
Oct 30 23:06:44.882783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.882743    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:06:44.882882 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.882828    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:06:44.882882 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:44.882861    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:06:45.153419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:45.153323    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" event={"ID":"79c1511e-3651-415c-a1f7-96b1d89b447a","Type":"ContainerStarted","Data":"cd0ddaf31ff0a18c9ee4d15f77d89b958ad99d9c7971fff02b08b4ba4d7c1d0c"}
Oct 30 23:06:45.153419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:45.153388    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" event={"ID":"79c1511e-3651-415c-a1f7-96b1d89b447a","Type":"ContainerStarted","Data":"b75e49fb4485599efd541985979548c955eba69820ff9b7d36373224b4dc09a4"}
Oct 30 23:06:45.203425 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:45.203376    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podStartSLOduration=1.203358991 podStartE2EDuration="1.203358991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:45.192689382 +0000 UTC m=+3589.231628269" watchObservedRunningTime="2026-10-30 23:06:45.203358991 +0000 UTC m=+3589.242297877"
Oct 30 23:06:45.560012 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:45.559923    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:45.560722 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:45.560641    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hcq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 23:06:45.560884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:45.560860    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podUID="79c1511e-3651-415c-a1f7-96b1d89b447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 23:06:46.563758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:46.563715    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hcq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 23:06:46.564437 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:46.564399    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podUID="79c1511e-3651-415c-a1f7-96b1d89b447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 23:06:47.558723 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:47.558681    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hcq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 23:06:47.559041 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:47.559010    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podUID="79c1511e-3651-415c-a1f7-96b1d89b447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 23:06:48.551796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:48.551752    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hcq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 23:06:48.552455 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:48.552416    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podUID="79c1511e-3651-415c-a1f7-96b1d89b447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 23:06:49.570245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:49.563670    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hcq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 23:06:49.570245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:49.563741    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podUID="79c1511e-3651-415c-a1f7-96b1d89b447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 23:06:50.557698 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:50.557654    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hcq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 23:06:50.558120 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:50.558091    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podUID="79c1511e-3651-415c-a1f7-96b1d89b447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 23:06:51.552236 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:51.552184    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hcq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused" start-of-body=
Oct 30 23:06:51.552726 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:51.552261    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podUID="79c1511e-3651-415c-a1f7-96b1d89b447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": dial tcp 10.133.1.146:15021: connect: connection refused"
Oct 30 23:06:53.098403 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:53.098365    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:53.098950 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:53.098928    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:53.099717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:53.099699    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62"
Oct 30 23:06:57.544992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:57.544960    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:06:57.553930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:06:57.553906    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:07:03.269838 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:03.269807    4581 scope.go:122] "RemoveContainer" containerID="bd9097e3cdcb612f48b3865c58651717fb994f63561cd17735b08590e42f8da2"
Oct 30 23:07:03.280203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:03.280180    4581 scope.go:122] "RemoveContainer" containerID="2550894cdda162550285e3e33829e022c24c3816c353c1d0dad88093a143ac63"
Oct 30 23:07:10.816525 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:07:10.816484    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:07:27.030499 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:27.030461    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"]
Oct 30 23:07:27.040578 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:27.040548    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bf421b-qsmck"]
Oct 30 23:07:28.096469 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:28.096434    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a61f0c-bcc3-41f8-93cf-91132f43567a" path="/var/lib/kubelet/pods/f9a61f0c-bcc3-41f8-93cf-91132f43567a/volumes"
Oct 30 23:07:31.027922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.027873    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"]
Oct 30 23:07:31.031665 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.031642    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.044496 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.044474    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:07:31.055815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.054628    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:07:31.055815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.054736    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:07:31.057921 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.057899    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"]
Oct 30 23:07:31.144916 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.144884    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:07:31.151378 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.151352    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02167675-af5c-478e-998b-1f0f496450ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.151467 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.151388    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02167675-af5c-478e-998b-1f0f496450ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.151571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.151551    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5f8\" (UniqueName: \"kubernetes.io/projected/02167675-af5c-478e-998b-1f0f496450ba-kube-api-access-6g5f8\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.252784 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.252745    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02167675-af5c-478e-998b-1f0f496450ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.253014 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.252833    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5f8\" (UniqueName: \"kubernetes.io/projected/02167675-af5c-478e-998b-1f0f496450ba-kube-api-access-6g5f8\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.253014 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.252890    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02167675-af5c-478e-998b-1f0f496450ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.253567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.253541    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02167675-af5c-478e-998b-1f0f496450ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.255413 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.255393    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02167675-af5c-478e-998b-1f0f496450ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.270138 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.270108    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5f8\" (UniqueName: \"kubernetes.io/projected/02167675-af5c-478e-998b-1f0f496450ba-kube-api-access-6g5f8\") pod \"maas-api-revoke-sub-c1e271a9-8dmv4\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.345259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.345135    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:31.651985 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:31.651960    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"]
Oct 30 23:07:31.654420 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:07:31.654388    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02167675_af5c_478e_998b_1f0f496450ba.slice/crio-43a5e4f443a553f272952785a34804c84fd3d3b9d1b940ea77ca592e18ba9fab WatchSource:0}: Error finding container 43a5e4f443a553f272952785a34804c84fd3d3b9d1b940ea77ca592e18ba9fab: Status 404 returned error can't find the container with id 43a5e4f443a553f272952785a34804c84fd3d3b9d1b940ea77ca592e18ba9fab
Oct 30 23:07:32.555112 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:32.555076    4581 generic.go:356] "Generic (PLEG): container finished" podID="02167675-af5c-478e-998b-1f0f496450ba" containerID="5878617809f80de65105f027f075949bf8277d2bcffab1d03e83f75d656213e2" exitCode=0
Oct 30 23:07:32.555638 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:32.555163    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4" event={"ID":"02167675-af5c-478e-998b-1f0f496450ba","Type":"ContainerDied","Data":"5878617809f80de65105f027f075949bf8277d2bcffab1d03e83f75d656213e2"}
Oct 30 23:07:32.555638 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:32.555232    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4" event={"ID":"02167675-af5c-478e-998b-1f0f496450ba","Type":"ContainerStarted","Data":"43a5e4f443a553f272952785a34804c84fd3d3b9d1b940ea77ca592e18ba9fab"}
Oct 30 23:07:33.705514 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.705486    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:33.777593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.777560    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02167675-af5c-478e-998b-1f0f496450ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02167675-af5c-478e-998b-1f0f496450ba-maas-api-xks-ca\") pod \"02167675-af5c-478e-998b-1f0f496450ba\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") "
Oct 30 23:07:33.777761 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.777605    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02167675-af5c-478e-998b-1f0f496450ba-kube-api-access-6g5f8\" (UniqueName: \"kubernetes.io/projected/02167675-af5c-478e-998b-1f0f496450ba-kube-api-access-6g5f8\") pod \"02167675-af5c-478e-998b-1f0f496450ba\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") "
Oct 30 23:07:33.777761 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.777665    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02167675-af5c-478e-998b-1f0f496450ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02167675-af5c-478e-998b-1f0f496450ba-maas-api-openshift-ca\") pod \"02167675-af5c-478e-998b-1f0f496450ba\" (UID: \"02167675-af5c-478e-998b-1f0f496450ba\") "
Oct 30 23:07:33.778107 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.778073    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02167675-af5c-478e-998b-1f0f496450ba-maas-api-openshift-ca" pod "02167675-af5c-478e-998b-1f0f496450ba" (UID: "02167675-af5c-478e-998b-1f0f496450ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:07:33.779857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.779830    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02167675-af5c-478e-998b-1f0f496450ba-maas-api-xks-ca" pod "02167675-af5c-478e-998b-1f0f496450ba" (UID: "02167675-af5c-478e-998b-1f0f496450ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:07:33.779977 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.779869    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02167675-af5c-478e-998b-1f0f496450ba-kube-api-access-6g5f8" pod "02167675-af5c-478e-998b-1f0f496450ba" (UID: "02167675-af5c-478e-998b-1f0f496450ba"). InnerVolumeSpecName "kube-api-access-6g5f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:07:33.878774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.878736    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02167675-af5c-478e-998b-1f0f496450ba-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:07:33.878774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.878771    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02167675-af5c-478e-998b-1f0f496450ba-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:07:33.878774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:33.878780    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g5f8\" (UniqueName: \"kubernetes.io/projected/02167675-af5c-478e-998b-1f0f496450ba-kube-api-access-6g5f8\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:07:34.576471 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:34.576438    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"
Oct 30 23:07:34.576687 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:34.576477    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4" event={"ID":"02167675-af5c-478e-998b-1f0f496450ba","Type":"ContainerDied","Data":"43a5e4f443a553f272952785a34804c84fd3d3b9d1b940ea77ca592e18ba9fab"}
Oct 30 23:07:34.576687 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:34.576527    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43a5e4f443a553f272952785a34804c84fd3d3b9d1b940ea77ca592e18ba9fab"
Oct 30 23:07:37.030806 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:37.030713    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"]
Oct 30 23:07:37.040613 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:37.040582    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1284f0d0-7sntv"]
Oct 30 23:07:38.094889 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:38.094854    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2654ae-cd6d-4714-9514-86cf17f9ef31" path="/var/lib/kubelet/pods/ad2654ae-cd6d-4714-9514-86cf17f9ef31/volumes"
Oct 30 23:07:40.815173 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:07:40.815136    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:07:44.391381 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.391343    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"]
Oct 30 23:07:44.392775 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.392745    4581 cpu_manager.go:414] "containerMap: removing container" podUID="02167675-af5c-478e-998b-1f0f496450ba" containerName="revoke-keys"
Oct 30 23:07:44.392942 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.392926    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02167675-af5c-478e-998b-1f0f496450ba" containerName="revoke-keys"
Oct 30 23:07:44.396677 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.396651    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="02167675-af5c-478e-998b-1f0f496450ba" containerName="revoke-keys"
Oct 30 23:07:44.401380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.401331    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.401720 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.401694    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"]
Oct 30 23:07:44.417671 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.410845    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.437005 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.435520    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"]
Oct 30 23:07:44.438665 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.438639    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"]
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482362    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482416    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482460    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482496    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b2d85e-38be-49c2-8ef6-31644a816dce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482527    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482554    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482581    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscld\" (UniqueName: \"kubernetes.io/projected/20b2d85e-38be-49c2-8ef6-31644a816dce-kube-api-access-rscld\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482634    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482662    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482706    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482748    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482793    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482823    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.482892    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.483001    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.483121    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlzb\" (UniqueName: \"kubernetes.io/projected/e50d3c0f-8e93-4e9b-9983-761f99918584-kube-api-access-xhlzb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.483152    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.483202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.483178    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50d3c0f-8e93-4e9b-9983-761f99918584-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.584788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.584711    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.584788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.584781    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.585175 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.584928    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.585175 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.584960    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.585175 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585005    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.585175 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585061    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.585175 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585129    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlzb\" (UniqueName: \"kubernetes.io/projected/e50d3c0f-8e93-4e9b-9983-761f99918584-kube-api-access-xhlzb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.585175 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585173    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585205    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50d3c0f-8e93-4e9b-9983-761f99918584-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585264    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585298    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585341    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585376    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b2d85e-38be-49c2-8ef6-31644a816dce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585407    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585434    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585465    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscld\" (UniqueName: \"kubernetes.io/projected/20b2d85e-38be-49c2-8ef6-31644a816dce-kube-api-access-rscld\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585528    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.585794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.585561    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.586497 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.586000    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.586497 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.586283    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.586849 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.586820    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.587504 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.587476    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.587685 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.587660    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b2d85e-38be-49c2-8ef6-31644a816dce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.588021 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.587995    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.588155 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.588131    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50d3c0f-8e93-4e9b-9983-761f99918584-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.588429 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.588408    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.588904 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.588878    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.589150 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.589094    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.591162 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.591137    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.591740 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.591686    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.593468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.593420    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.593624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.593600    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.612518 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.612152    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlzb\" (UniqueName: \"kubernetes.io/projected/e50d3c0f-8e93-4e9b-9983-761f99918584-kube-api-access-xhlzb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.613953 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.613925    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b2d85e-38be-49c2-8ef6-31644a816dce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.614759 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.614659    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50d3c0f-8e93-4e9b-9983-761f99918584-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tbk5w\" (UID: \"e50d3c0f-8e93-4e9b-9983-761f99918584\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:44.614759 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.614716    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscld\" (UniqueName: \"kubernetes.io/projected/20b2d85e-38be-49c2-8ef6-31644a816dce-kube-api-access-rscld\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h8mv5\" (UID: \"20b2d85e-38be-49c2-8ef6-31644a816dce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.730000 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.729919    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:44.738452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:44.738427    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:45.060254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.060034    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"]
Oct 30 23:07:45.062692 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:07:45.062655    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b2d85e_38be_49c2_8ef6_31644a816dce.slice/crio-e2671c6e5692a6694fac7daf621e73a7c0c8481f753cad5b8ef11121665063c4 WatchSource:0}: Error finding container e2671c6e5692a6694fac7daf621e73a7c0c8481f753cad5b8ef11121665063c4: Status 404 returned error can't find the container with id e2671c6e5692a6694fac7daf621e73a7c0c8481f753cad5b8ef11121665063c4
Oct 30 23:07:45.064864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.064830    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:07:45.064959 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.064895    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:07:45.064959 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.064947    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:07:45.075091 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.075067    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"]
Oct 30 23:07:45.076151 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:07:45.076124    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50d3c0f_8e93_4e9b_9983_761f99918584.slice/crio-096e85f70ae2746711f0616909ca292b699a169342c3bfeb1debb6dab28e0125 WatchSource:0}: Error finding container 096e85f70ae2746711f0616909ca292b699a169342c3bfeb1debb6dab28e0125: Status 404 returned error can't find the container with id 096e85f70ae2746711f0616909ca292b699a169342c3bfeb1debb6dab28e0125
Oct 30 23:07:45.078287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.078256    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:07:45.078392 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.078317    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:07:45.078392 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.078351    4581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 23:07:45.652409 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.652366    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" event={"ID":"e50d3c0f-8e93-4e9b-9983-761f99918584","Type":"ContainerStarted","Data":"ae56297b5581880522b0f5c965a982a19f495056911a1e9b1bbe7f404b5919b7"}
Oct 30 23:07:45.661491 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.661445    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" event={"ID":"e50d3c0f-8e93-4e9b-9983-761f99918584","Type":"ContainerStarted","Data":"096e85f70ae2746711f0616909ca292b699a169342c3bfeb1debb6dab28e0125"}
Oct 30 23:07:45.670252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.669380    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" event={"ID":"20b2d85e-38be-49c2-8ef6-31644a816dce","Type":"ContainerStarted","Data":"e95f018e7a1309ab48a523419b5f9cfeaa91d1fc69f5250ad54977faffe0f219"}
Oct 30 23:07:45.670252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.669453    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" event={"ID":"20b2d85e-38be-49c2-8ef6-31644a816dce","Type":"ContainerStarted","Data":"e2671c6e5692a6694fac7daf621e73a7c0c8481f753cad5b8ef11121665063c4"}
Oct 30 23:07:45.730336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.730299    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:45.738575 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.738548    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:45.774241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.774166    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podStartSLOduration=1.774134924 podStartE2EDuration="1.774134924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:45.772263505 +0000 UTC m=+3649.811202395" watchObservedRunningTime="2026-10-30 23:07:45.774134924 +0000 UTC m=+3649.813073811"
Oct 30 23:07:45.779258 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:45.775252    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podStartSLOduration=1.775208205 podStartE2EDuration="1.775208205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:45.726258547 +0000 UTC m=+3649.765197435" watchObservedRunningTime="2026-10-30 23:07:45.775208205 +0000 UTC m=+3649.814147092"
Oct 30 23:07:46.732315 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:46.732278    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused" start-of-body=
Oct 30 23:07:46.732723 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:46.732332    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused"
Oct 30 23:07:46.740176 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:46.740147    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body=
Oct 30 23:07:46.740369 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:46.740188    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused"
Oct 30 23:07:47.730510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:47.730473    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused" start-of-body=
Oct 30 23:07:47.730698 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:47.730529    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused"
Oct 30 23:07:47.739465 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:47.739440    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body=
Oct 30 23:07:47.739822 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:47.739486    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused"
Oct 30 23:07:48.027704 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:48.027620    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"]
Oct 30 23:07:48.037183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:48.037154    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3252ab44-pcgx9"]
Oct 30 23:07:48.127254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:48.125044    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8655d1-7428-4ca1-b6ce-351692ba1815" path="/var/lib/kubelet/pods/ae8655d1-7428-4ca1-b6ce-351692ba1815/volumes"
Oct 30 23:07:48.730577 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:48.730541    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused" start-of-body=
Oct 30 23:07:48.730915 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:48.730867    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused"
Oct 30 23:07:48.738958 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:48.738928    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body=
Oct 30 23:07:48.739113 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:48.738981    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused"
Oct 30 23:07:49.732008 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:49.731968    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused" start-of-body=
Oct 30 23:07:49.732720 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:49.732689    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused"
Oct 30 23:07:49.740317 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:49.740286    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body=
Oct 30 23:07:49.740560 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:49.740534    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused"
Oct 30 23:07:50.756260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:50.743620    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused" start-of-body=
Oct 30 23:07:50.756260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:50.743684    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused"
Oct 30 23:07:50.756260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:50.744583    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body=
Oct 30 23:07:50.756260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:50.744625    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused"
Oct 30 23:07:51.730713 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:51.730670    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused" start-of-body=
Oct 30 23:07:51.730893 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:51.730739    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused"
Oct 30 23:07:51.739127 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:51.739092    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body=
Oct 30 23:07:51.739346 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:51.739151    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused"
Oct 30 23:07:52.730554 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:52.730514    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused" start-of-body=
Oct 30 23:07:52.731007 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:52.730587    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": dial tcp 10.133.1.148:15021: connect: connection refused"
Oct 30 23:07:52.739007 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:52.738975    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused" start-of-body=
Oct 30 23:07:52.739100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:52.739040    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": dial tcp 10.133.1.149:15021: connect: connection refused"
Oct 30 23:07:54.304918 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.304872    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"]
Oct 30 23:07:54.309868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.309844    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.329082 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.329046    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:07:54.329289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.329137    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:07:54.332615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.332588    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltcw\" (UniqueName: \"kubernetes.io/projected/984c5fcd-1d88-4850-bfb0-0239f65fa2af-kube-api-access-xltcw\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.332769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.332630    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.332769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.332757    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.347104 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.347061    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"]
Oct 30 23:07:54.351657 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.351631    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:07:54.351796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.351689    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:07:54.434460 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.434418    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltcw\" (UniqueName: \"kubernetes.io/projected/984c5fcd-1d88-4850-bfb0-0239f65fa2af-kube-api-access-xltcw\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.434660 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.434505    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.434660 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.434624    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.435307 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.435278    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.437073 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.437051    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.462161 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.462128    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltcw\" (UniqueName: \"kubernetes.io/projected/984c5fcd-1d88-4850-bfb0-0239f65fa2af-kube-api-access-xltcw\") pod \"maas-api-revoke-sub-2e1f5b5f-rvhn8\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.623687 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.623640    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:54.779364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.778610    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:54.779364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.778692    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:54.783046 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.781472    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:54.783046 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.781539    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:54.968659 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:54.968549    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"]
Oct 30 23:07:55.820510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:55.820406    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:55.820510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:55.820476    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:55.823362 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:55.823339    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:55.823435 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:55.823379    4581 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:55.873643 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:55.873589    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8" event={"ID":"984c5fcd-1d88-4850-bfb0-0239f65fa2af","Type":"ContainerStarted","Data":"11c00c1b313c8be685114481fd2f7e8ee85d9aec66e2e63eb8bdaaca0216906b"}
Oct 30 23:07:55.873942 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:55.873915    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8" event={"ID":"984c5fcd-1d88-4850-bfb0-0239f65fa2af","Type":"ContainerStarted","Data":"d60bc4cf328f4b034246b60e340dd117b66ef9a19e58637285c3645f6599274e"}
Oct 30 23:07:55.924388 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:55.924297    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8" podStartSLOduration=1.924276828 podStartE2EDuration="1.924276828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:55.921916926 +0000 UTC m=+3659.960855812" watchObservedRunningTime="2026-10-30 23:07:55.924276828 +0000 UTC m=+3659.963215715"
Oct 30 23:07:56.203694 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:56.203660    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:56.204146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:56.204125    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:56.205246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:56.205188    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w"
Oct 30 23:07:56.207122 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:56.207101    4581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:56.207524 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:56.207499    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:56.208186 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:56.208170    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5"
Oct 30 23:07:56.881872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:56.881836    4581 generic.go:356] "Generic (PLEG): container finished" podID="984c5fcd-1d88-4850-bfb0-0239f65fa2af" containerID="11c00c1b313c8be685114481fd2f7e8ee85d9aec66e2e63eb8bdaaca0216906b" exitCode=0
Oct 30 23:07:56.882306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:56.881929    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8" event={"ID":"984c5fcd-1d88-4850-bfb0-0239f65fa2af","Type":"ContainerDied","Data":"11c00c1b313c8be685114481fd2f7e8ee85d9aec66e2e63eb8bdaaca0216906b"}
Oct 30 23:07:58.031868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.031838    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:58.070650 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.070613    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-xks-ca\") pod \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") "
Oct 30 23:07:58.070847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.070696    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-openshift-ca\") pod \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") "
Oct 30 23:07:58.070847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.070730    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984c5fcd-1d88-4850-bfb0-0239f65fa2af-kube-api-access-xltcw\" (UniqueName: \"kubernetes.io/projected/984c5fcd-1d88-4850-bfb0-0239f65fa2af-kube-api-access-xltcw\") pod \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\" (UID: \"984c5fcd-1d88-4850-bfb0-0239f65fa2af\") "
Oct 30 23:07:58.071552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.071522    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-openshift-ca" pod "984c5fcd-1d88-4850-bfb0-0239f65fa2af" (UID: "984c5fcd-1d88-4850-bfb0-0239f65fa2af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:07:58.073555 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.073523    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984c5fcd-1d88-4850-bfb0-0239f65fa2af-kube-api-access-xltcw" pod "984c5fcd-1d88-4850-bfb0-0239f65fa2af" (UID: "984c5fcd-1d88-4850-bfb0-0239f65fa2af"). InnerVolumeSpecName "kube-api-access-xltcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:07:58.074635 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.074594    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-xks-ca" pod "984c5fcd-1d88-4850-bfb0-0239f65fa2af" (UID: "984c5fcd-1d88-4850-bfb0-0239f65fa2af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:07:58.171983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.171907    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:07:58.171983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.171932    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c5fcd-1d88-4850-bfb0-0239f65fa2af-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:07:58.171983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.171943    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xltcw\" (UniqueName: \"kubernetes.io/projected/984c5fcd-1d88-4850-bfb0-0239f65fa2af-kube-api-access-xltcw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:07:58.897994 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.897954    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"
Oct 30 23:07:58.898172 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.897986    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8" event={"ID":"984c5fcd-1d88-4850-bfb0-0239f65fa2af","Type":"ContainerDied","Data":"d60bc4cf328f4b034246b60e340dd117b66ef9a19e58637285c3645f6599274e"}
Oct 30 23:07:58.898172 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:58.898037    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60bc4cf328f4b034246b60e340dd117b66ef9a19e58637285c3645f6599274e"
Oct 30 23:07:59.029973 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:59.029931    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"]
Oct 30 23:07:59.033022 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:07:59.032997    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2c776d-9p6sf"]
Oct 30 23:08:00.104279 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:00.104236    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdc3870-d2c2-400b-a663-8caaf51e37e2" path="/var/lib/kubelet/pods/ebdc3870-d2c2-400b-a663-8caaf51e37e2/volumes"
Oct 30 23:08:03.316209 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:03.316178    4581 scope.go:122] "RemoveContainer" containerID="2b01c9075fff44d2fcd3c01445cbf1cbab2cbb71c5c929d2d35b4dfdb8f0f2c9"
Oct 30 23:08:03.326635 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:03.326613    4581 scope.go:122] "RemoveContainer" containerID="5e1b48a2ae5caf061cf6d266bfe1e03172a9021dc282314afb419ae1a7a184a5"
Oct 30 23:08:03.336203 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:03.336178    4581 scope.go:122] "RemoveContainer" containerID="06b4ebf54b9f62c3b1d5878ba2983312326a5a81e2696acd5d4e073ea4f6cee5"
Oct 30 23:08:03.345446 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:03.345424    4581 scope.go:122] "RemoveContainer" containerID="5d0e3d379f7aaa32bb2b5ff10e014d90aaa657c7325b8ab2ad445b8e75deff05"
Oct 30 23:08:05.570382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.570343    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"]
Oct 30 23:08:05.570992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.570749    4581 cpu_manager.go:414] "containerMap: removing container" podUID="984c5fcd-1d88-4850-bfb0-0239f65fa2af" containerName="revoke-keys"
Oct 30 23:08:05.570992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.570759    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984c5fcd-1d88-4850-bfb0-0239f65fa2af" containerName="revoke-keys"
Oct 30 23:08:05.570992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.570821    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="984c5fcd-1d88-4850-bfb0-0239f65fa2af" containerName="revoke-keys"
Oct 30 23:08:05.575766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.575745    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.601484 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.601453    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:08:05.601484 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.601481    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:08:05.601730 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.601512    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:08:05.601900 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.601879    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:08:05.605422 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.605397    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"]
Oct 30 23:08:05.645011 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.644982    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.645199 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.645024    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.645199 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.645127    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgcft\" (UniqueName: \"kubernetes.io/projected/71a27700-0280-4b3e-8dda-4e8e457bfd8b-kube-api-access-tgcft\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.746681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.746640    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.746681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.746687    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.746965 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.746713    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcft\" (UniqueName: \"kubernetes.io/projected/71a27700-0280-4b3e-8dda-4e8e457bfd8b-kube-api-access-tgcft\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.747350 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.747330    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.749284 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.749260    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.769909 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.769879    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcft\" (UniqueName: \"kubernetes.io/projected/71a27700-0280-4b3e-8dda-4e8e457bfd8b-kube-api-access-tgcft\") pod \"maas-api-revoke-sub-ec843018-sr2xw\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:05.892868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:05.892830    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:06.199014 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:06.198985    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"]
Oct 30 23:08:06.202505 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:08:06.202472    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a27700_0280_4b3e_8dda_4e8e457bfd8b.slice/crio-1b9cdfae2441537e9b2b1430f9f705ac16cc0e538fc7efd2456628abbcb5b938 WatchSource:0}: Error finding container 1b9cdfae2441537e9b2b1430f9f705ac16cc0e538fc7efd2456628abbcb5b938: Status 404 returned error can't find the container with id 1b9cdfae2441537e9b2b1430f9f705ac16cc0e538fc7efd2456628abbcb5b938
Oct 30 23:08:06.968357 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:06.968291    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw" event={"ID":"71a27700-0280-4b3e-8dda-4e8e457bfd8b","Type":"ContainerStarted","Data":"312fb099d11d590e25c8140276e0fcfbcfd8dc01abdaab800571dab5c4914d91"}
Oct 30 23:08:06.968948 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:06.968433    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw" event={"ID":"71a27700-0280-4b3e-8dda-4e8e457bfd8b","Type":"ContainerStarted","Data":"1b9cdfae2441537e9b2b1430f9f705ac16cc0e538fc7efd2456628abbcb5b938"}
Oct 30 23:08:07.013783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:07.013705    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw" podStartSLOduration=2.013683038 podStartE2EDuration="2.013683038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:07.005443517 +0000 UTC m=+3671.044382404" watchObservedRunningTime="2026-10-30 23:08:07.013683038 +0000 UTC m=+3671.052621925"
Oct 30 23:08:08.005023 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:08.004975    4581 generic.go:356] "Generic (PLEG): container finished" podID="71a27700-0280-4b3e-8dda-4e8e457bfd8b" containerID="312fb099d11d590e25c8140276e0fcfbcfd8dc01abdaab800571dab5c4914d91" exitCode=0
Oct 30 23:08:08.005594 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:08.005267    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw" event={"ID":"71a27700-0280-4b3e-8dda-4e8e457bfd8b","Type":"ContainerDied","Data":"312fb099d11d590e25c8140276e0fcfbcfd8dc01abdaab800571dab5c4914d91"}
Oct 30 23:08:09.162672 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.162644    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:09.304698 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.304612    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a27700-0280-4b3e-8dda-4e8e457bfd8b-kube-api-access-tgcft\" (UniqueName: \"kubernetes.io/projected/71a27700-0280-4b3e-8dda-4e8e457bfd8b-kube-api-access-tgcft\") pod \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") "
Oct 30 23:08:09.304870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.304731    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-xks-ca\") pod \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") "
Oct 30 23:08:09.304870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.304759    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-openshift-ca\") pod \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\" (UID: \"71a27700-0280-4b3e-8dda-4e8e457bfd8b\") "
Oct 30 23:08:09.305157 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.305128    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-openshift-ca" pod "71a27700-0280-4b3e-8dda-4e8e457bfd8b" (UID: "71a27700-0280-4b3e-8dda-4e8e457bfd8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:08:09.306961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.306926    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a27700-0280-4b3e-8dda-4e8e457bfd8b-kube-api-access-tgcft" pod "71a27700-0280-4b3e-8dda-4e8e457bfd8b" (UID: "71a27700-0280-4b3e-8dda-4e8e457bfd8b"). InnerVolumeSpecName "kube-api-access-tgcft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:08:09.307070 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.306955    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-xks-ca" pod "71a27700-0280-4b3e-8dda-4e8e457bfd8b" (UID: "71a27700-0280-4b3e-8dda-4e8e457bfd8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:08:09.406699 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.406660    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgcft\" (UniqueName: \"kubernetes.io/projected/71a27700-0280-4b3e-8dda-4e8e457bfd8b-kube-api-access-tgcft\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:09.406699 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.406693    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:09.406934 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:09.406703    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a27700-0280-4b3e-8dda-4e8e457bfd8b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:10.021938 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:10.021900    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"
Oct 30 23:08:10.021938 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:10.021923    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw" event={"ID":"71a27700-0280-4b3e-8dda-4e8e457bfd8b","Type":"ContainerDied","Data":"1b9cdfae2441537e9b2b1430f9f705ac16cc0e538fc7efd2456628abbcb5b938"}
Oct 30 23:08:10.022184 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:10.021970    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b9cdfae2441537e9b2b1430f9f705ac16cc0e538fc7efd2456628abbcb5b938"
Oct 30 23:08:10.817322 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:08:10.817283    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:08:24.145245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:24.144353    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hcq62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:08:24.145245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:24.144456    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hcq62" podUID="79c1511e-3651-415c-a1f7-96b1d89b447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:08:33.433892 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.433856    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"]
Oct 30 23:08:33.434347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.434288    4581 cpu_manager.go:414] "containerMap: removing container" podUID="71a27700-0280-4b3e-8dda-4e8e457bfd8b" containerName="revoke-keys"
Oct 30 23:08:33.434347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.434301    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a27700-0280-4b3e-8dda-4e8e457bfd8b" containerName="revoke-keys"
Oct 30 23:08:33.434484 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.434378    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a27700-0280-4b3e-8dda-4e8e457bfd8b" containerName="revoke-keys"
Oct 30 23:08:33.437636 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.437615    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.450510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.450480    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:08:33.450510 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.450489    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:08:33.460537 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.460510    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:08:33.460719 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.460703    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:08:33.464147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.464114    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"]
Oct 30 23:08:33.493594 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.493551    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-kube-api-access-mprng\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.493784 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.493752    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.493921 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.493896    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.594818 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.594776    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.595008 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.594845    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.595008 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.594877    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-kube-api-access-mprng\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.595481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.595454    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.597280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.597262    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.612909 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.612885    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-kube-api-access-mprng\") pod \"maas-api-revoke-sub-046e1343-s6mzl\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:33.750715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:33.750633    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:34.053382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:34.053353    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"]
Oct 30 23:08:34.055063 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:08:34.055036    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d10d90_8815_4f2e_ba2c_a55e5cb0bfdb.slice/crio-8eb8b9a4d8d2a913788d9226c75b4e4bdb6b6bde7d6c1898a22c0999cb7a8266 WatchSource:0}: Error finding container 8eb8b9a4d8d2a913788d9226c75b4e4bdb6b6bde7d6c1898a22c0999cb7a8266: Status 404 returned error can't find the container with id 8eb8b9a4d8d2a913788d9226c75b4e4bdb6b6bde7d6c1898a22c0999cb7a8266
Oct 30 23:08:34.256076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:34.256041    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl" event={"ID":"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb","Type":"ContainerStarted","Data":"8eb8b9a4d8d2a913788d9226c75b4e4bdb6b6bde7d6c1898a22c0999cb7a8266"}
Oct 30 23:08:35.264152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:35.264056    4581 generic.go:356] "Generic (PLEG): container finished" podID="a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb" containerID="03610b0e060987035a2048cd86364e7707be96be13bcf50cfbc69e24ee6ec8fd" exitCode=0
Oct 30 23:08:35.264152 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:35.264087    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl" event={"ID":"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb","Type":"ContainerDied","Data":"03610b0e060987035a2048cd86364e7707be96be13bcf50cfbc69e24ee6ec8fd"}
Oct 30 23:08:36.416499 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.416473    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:36.523887 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.523792    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-openshift-ca\") pod \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") "
Oct 30 23:08:36.523887 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.523856    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-xks-ca\") pod \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") "
Oct 30 23:08:36.524147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.523952    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-kube-api-access-mprng\") pod \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\" (UID: \"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb\") "
Oct 30 23:08:36.524285 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.524246    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-openshift-ca" pod "a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb" (UID: "a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:08:36.526109 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.526084    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-xks-ca" pod "a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb" (UID: "a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:08:36.526201 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.526175    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-kube-api-access-mprng" pod "a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb" (UID: "a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb"). InnerVolumeSpecName "kube-api-access-mprng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:08:36.625611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.625579    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:36.625611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.625610    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mprng\" (UniqueName: \"kubernetes.io/projected/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-kube-api-access-mprng\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:36.625611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:36.625619    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:37.279123 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:37.279091    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"
Oct 30 23:08:37.279448 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:37.279126    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl" event={"ID":"a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb","Type":"ContainerDied","Data":"8eb8b9a4d8d2a913788d9226c75b4e4bdb6b6bde7d6c1898a22c0999cb7a8266"}
Oct 30 23:08:37.279448 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:37.279173    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb8b9a4d8d2a913788d9226c75b4e4bdb6b6bde7d6c1898a22c0999cb7a8266"
Oct 30 23:08:40.816398 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:08:40.816359    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:08:43.783468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.783433    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"]
Oct 30 23:08:43.783910 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.783895    4581 cpu_manager.go:414] "containerMap: removing container" podUID="a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb" containerName="revoke-keys"
Oct 30 23:08:43.783958 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.783913    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb" containerName="revoke-keys"
Oct 30 23:08:43.784020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.784010    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb" containerName="revoke-keys"
Oct 30 23:08:43.789025 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.789003    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:43.801907 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.801877    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:08:43.811693 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.811666    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:08:43.817792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.817758    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"]
Oct 30 23:08:43.832246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.832203    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:08:43.832478 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.832245    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:08:43.897877 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.897839    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:43.897877 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.897876    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:43.898113 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.897914    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gc9t\" (UniqueName: \"kubernetes.io/projected/e1151835-e326-44c0-a38e-bc2acb33293c-kube-api-access-9gc9t\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:43.999158 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.999118    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:43.999158 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.999166    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:43.999477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.999248    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gc9t\" (UniqueName: \"kubernetes.io/projected/e1151835-e326-44c0-a38e-bc2acb33293c-kube-api-access-9gc9t\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:43.999840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:43.999819    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:44.001934 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:44.001911    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:44.017350 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:44.017316    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gc9t\" (UniqueName: \"kubernetes.io/projected/e1151835-e326-44c0-a38e-bc2acb33293c-kube-api-access-9gc9t\") pod \"maas-api-revoke-sub-d3fba056-74tkr\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:44.108347 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:44.108318    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:44.416570 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:44.416537    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"]
Oct 30 23:08:44.418467 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:08:44.418435    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1151835_e326_44c0_a38e_bc2acb33293c.slice/crio-6bbcef9066b5a1517c2f88d8c458ac9a10d8e4a3f8ac0a09d6429973536fe5b4 WatchSource:0}: Error finding container 6bbcef9066b5a1517c2f88d8c458ac9a10d8e4a3f8ac0a09d6429973536fe5b4: Status 404 returned error can't find the container with id 6bbcef9066b5a1517c2f88d8c458ac9a10d8e4a3f8ac0a09d6429973536fe5b4
Oct 30 23:08:45.356657 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:45.356617    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr" event={"ID":"e1151835-e326-44c0-a38e-bc2acb33293c","Type":"ContainerStarted","Data":"e28d49feb73782b17946930ebeaa40fbe2b5d2168bd63a3dd7b2f5b8a436c57c"}
Oct 30 23:08:45.357493 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:45.357288    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr" event={"ID":"e1151835-e326-44c0-a38e-bc2acb33293c","Type":"ContainerStarted","Data":"6bbcef9066b5a1517c2f88d8c458ac9a10d8e4a3f8ac0a09d6429973536fe5b4"}
Oct 30 23:08:45.401096 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:45.401032    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr" podStartSLOduration=2.4010111849999998 podStartE2EDuration="2.401011185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:45.394660814 +0000 UTC m=+3709.433599701" watchObservedRunningTime="2026-10-30 23:08:45.401011185 +0000 UTC m=+3709.439950072"
Oct 30 23:08:46.423796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:46.423072    4581 generic.go:356] "Generic (PLEG): container finished" podID="e1151835-e326-44c0-a38e-bc2acb33293c" containerID="e28d49feb73782b17946930ebeaa40fbe2b5d2168bd63a3dd7b2f5b8a436c57c" exitCode=0
Oct 30 23:08:46.423796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:46.423393    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr" event={"ID":"e1151835-e326-44c0-a38e-bc2acb33293c","Type":"ContainerDied","Data":"e28d49feb73782b17946930ebeaa40fbe2b5d2168bd63a3dd7b2f5b8a436c57c"}
Oct 30 23:08:47.868247 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:47.860365    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:08:47.868247 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:47.860505    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:08:48.146741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.146711    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:48.242486 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.242437    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-xks-ca\") pod \"e1151835-e326-44c0-a38e-bc2acb33293c\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") "
Oct 30 23:08:48.242658 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.242581    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1151835-e326-44c0-a38e-bc2acb33293c-kube-api-access-9gc9t\" (UniqueName: \"kubernetes.io/projected/e1151835-e326-44c0-a38e-bc2acb33293c-kube-api-access-9gc9t\") pod \"e1151835-e326-44c0-a38e-bc2acb33293c\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") "
Oct 30 23:08:48.242716 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.242654    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-openshift-ca\") pod \"e1151835-e326-44c0-a38e-bc2acb33293c\" (UID: \"e1151835-e326-44c0-a38e-bc2acb33293c\") "
Oct 30 23:08:48.243028 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.242995    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-openshift-ca" pod "e1151835-e326-44c0-a38e-bc2acb33293c" (UID: "e1151835-e326-44c0-a38e-bc2acb33293c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:08:48.244830 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.244803    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-xks-ca" pod "e1151835-e326-44c0-a38e-bc2acb33293c" (UID: "e1151835-e326-44c0-a38e-bc2acb33293c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:08:48.244933 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.244868    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1151835-e326-44c0-a38e-bc2acb33293c-kube-api-access-9gc9t" pod "e1151835-e326-44c0-a38e-bc2acb33293c" (UID: "e1151835-e326-44c0-a38e-bc2acb33293c"). InnerVolumeSpecName "kube-api-access-9gc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:08:48.344026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.343917    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gc9t\" (UniqueName: \"kubernetes.io/projected/e1151835-e326-44c0-a38e-bc2acb33293c-kube-api-access-9gc9t\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:48.344026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.343974    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:48.344026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.343986    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1151835-e326-44c0-a38e-bc2acb33293c-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:08:48.493634 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.493602    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"
Oct 30 23:08:48.493825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.493642    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr" event={"ID":"e1151835-e326-44c0-a38e-bc2acb33293c","Type":"ContainerDied","Data":"6bbcef9066b5a1517c2f88d8c458ac9a10d8e4a3f8ac0a09d6429973536fe5b4"}
Oct 30 23:08:48.493825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:48.493689    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bbcef9066b5a1517c2f88d8c458ac9a10d8e4a3f8ac0a09d6429973536fe5b4"
Oct 30 23:08:53.028911 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:53.028879    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"]
Oct 30 23:08:53.038718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:53.038688    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"]
Oct 30 23:08:53.042035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:53.042007    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2446288b-94cfn"]
Oct 30 23:08:53.044684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:53.044666    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0ccdb8-mqk4k"]
Oct 30 23:08:54.096502 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:54.096468    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62bb59d5-ee90-4095-8b52-c956cb3fff9f" path="/var/lib/kubelet/pods/62bb59d5-ee90-4095-8b52-c956cb3fff9f/volumes"
Oct 30 23:08:54.096909 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:08:54.096884    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a7548e-51bd-4686-a413-a3f0e75306f6" path="/var/lib/kubelet/pods/87a7548e-51bd-4686-a413-a3f0e75306f6/volumes"
Oct 30 23:09:02.845122 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:02.845061    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:09:02.845861 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:02.845828    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:09:03.387286 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.387247    4581 scope.go:122] "RemoveContainer" containerID="154db057e611dd26637248bbd1e4e3f3f1afd2e27ab61ee283937fe79e414257"
Oct 30 23:09:03.401885 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.401859    4581 scope.go:122] "RemoveContainer" containerID="bd98eb59fbcec91dcc8dff50b5df5fef9af74c736944efade192a76009ae772b"
Oct 30 23:09:03.411789 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.411766    4581 scope.go:122] "RemoveContainer" containerID="21a2e264da0ce4f2b29bbbdbe093ca25cc1a89ad1cb70ca1925d7f3dcf382911"
Oct 30 23:09:03.421707 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.421681    4581 scope.go:122] "RemoveContainer" containerID="389a15353a1e13219db70b18a307d5bbc8e3c149467e82ca75fad2d94d67f352"
Oct 30 23:09:03.433301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.433272    4581 scope.go:122] "RemoveContainer" containerID="e879d53ad563a70f12bfef12a9ba37e861adb326a9dbbe1cf3ba1253f0800950"
Oct 30 23:09:03.444687 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.444663    4581 scope.go:122] "RemoveContainer" containerID="9e025cfbd32d81d5375e90829b8126d9adaeb9e5285070abe64c4a04b1ada34e"
Oct 30 23:09:03.454151 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.454130    4581 scope.go:122] "RemoveContainer" containerID="51aa94efb9f79980efa32b2001ad45f6c61958a44cd7f19171bdbd051d33e8a4"
Oct 30 23:09:03.463399 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.463376    4581 scope.go:122] "RemoveContainer" containerID="aeed745158c006c1e978acf03a7437812c65aa010a6514466419f434a826758a"
Oct 30 23:09:03.472320 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:03.472298    4581 scope.go:122] "RemoveContainer" containerID="348b0ab9354fa68bfa1e3a00f89426023542e5cd414c5c0b4e8fd4bc9c722f16"
Oct 30 23:09:05.247466 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.247431    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"]
Oct 30 23:09:05.247885 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.247831    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e1151835-e326-44c0-a38e-bc2acb33293c" containerName="revoke-keys"
Oct 30 23:09:05.247885 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.247843    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1151835-e326-44c0-a38e-bc2acb33293c" containerName="revoke-keys"
Oct 30 23:09:05.247972 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.247938    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1151835-e326-44c0-a38e-bc2acb33293c" containerName="revoke-keys"
Oct 30 23:09:05.253415 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.253396    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.271241 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.271179    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"]
Oct 30 23:09:05.277060 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.277011    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:09:05.277060 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.277015    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:09:05.277060 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.277054    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:09:05.277325 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.277102    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:09:05.351563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.351529    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.351756 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.351594    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48snd\" (UniqueName: \"kubernetes.io/projected/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-kube-api-access-48snd\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.351756 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.351709    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.452653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.452622    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.452881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.452686    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.452881 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.452731    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48snd\" (UniqueName: \"kubernetes.io/projected/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-kube-api-access-48snd\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.453414 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.453391    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.455187 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.455164    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.470428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.470401    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48snd\" (UniqueName: \"kubernetes.io/projected/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-kube-api-access-48snd\") pod \"maas-api-revoke-sub-ae594979-2vnxc\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.566140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.566055    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:05.866767 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:05.866733    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"]
Oct 30 23:09:05.870636 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:09:05.870607    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7badff_2f63_4a26_ba3e_87ad50fd4d18.slice/crio-c170405c1a513492fa344b1b84526e86305603c20242d5c3772db92b8da1b6e9 WatchSource:0}: Error finding container c170405c1a513492fa344b1b84526e86305603c20242d5c3772db92b8da1b6e9: Status 404 returned error can't find the container with id c170405c1a513492fa344b1b84526e86305603c20242d5c3772db92b8da1b6e9
Oct 30 23:09:06.029126 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:06.029094    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"]
Oct 30 23:09:06.038580 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:06.034983    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60728c12-szf49"]
Oct 30 23:09:06.095146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:06.095112    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81dfe808-2ff0-4d65-b910-50d112f89026" path="/var/lib/kubelet/pods/81dfe808-2ff0-4d65-b910-50d112f89026/volumes"
Oct 30 23:09:06.671255 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:06.671199    4581 generic.go:356] "Generic (PLEG): container finished" podID="6c7badff-2f63-4a26-ba3e-87ad50fd4d18" containerID="e5f162049bbbd55b28052ce1ba6569c49b1918a92acfb33d3492da6a04ad3436" exitCode=0
Oct 30 23:09:06.671600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:06.671256    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc" event={"ID":"6c7badff-2f63-4a26-ba3e-87ad50fd4d18","Type":"ContainerDied","Data":"e5f162049bbbd55b28052ce1ba6569c49b1918a92acfb33d3492da6a04ad3436"}
Oct 30 23:09:06.671600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:06.671306    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc" event={"ID":"6c7badff-2f63-4a26-ba3e-87ad50fd4d18","Type":"ContainerStarted","Data":"c170405c1a513492fa344b1b84526e86305603c20242d5c3772db92b8da1b6e9"}
Oct 30 23:09:07.028295 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.028188    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"]
Oct 30 23:09:07.031144 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.031120    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7acc14d9-gvwcs"]
Oct 30 23:09:07.824691 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.824663    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:07.979131 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.979042    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-xks-ca\") pod \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") "
Oct 30 23:09:07.979344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.979167    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-openshift-ca\") pod \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") "
Oct 30 23:09:07.979344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.979197    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-kube-api-access-48snd\" (UniqueName: \"kubernetes.io/projected/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-kube-api-access-48snd\") pod \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\" (UID: \"6c7badff-2f63-4a26-ba3e-87ad50fd4d18\") "
Oct 30 23:09:07.979570 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.979538    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-openshift-ca" pod "6c7badff-2f63-4a26-ba3e-87ad50fd4d18" (UID: "6c7badff-2f63-4a26-ba3e-87ad50fd4d18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:09:07.981404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.981377    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-xks-ca" pod "6c7badff-2f63-4a26-ba3e-87ad50fd4d18" (UID: "6c7badff-2f63-4a26-ba3e-87ad50fd4d18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:09:07.981514 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:07.981493    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-kube-api-access-48snd" pod "6c7badff-2f63-4a26-ba3e-87ad50fd4d18" (UID: "6c7badff-2f63-4a26-ba3e-87ad50fd4d18"). InnerVolumeSpecName "kube-api-access-48snd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:09:08.080010 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:08.079975    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:08.080010 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:08.080007    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48snd\" (UniqueName: \"kubernetes.io/projected/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-kube-api-access-48snd\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:08.080249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:08.080023    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7badff-2f63-4a26-ba3e-87ad50fd4d18-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:08.096038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:08.096005    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab171e36-3a54-4c31-bd6a-5d179092b14a" path="/var/lib/kubelet/pods/ab171e36-3a54-4c31-bd6a-5d179092b14a/volumes"
Oct 30 23:09:08.686717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:08.686674    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc" event={"ID":"6c7badff-2f63-4a26-ba3e-87ad50fd4d18","Type":"ContainerDied","Data":"c170405c1a513492fa344b1b84526e86305603c20242d5c3772db92b8da1b6e9"}
Oct 30 23:09:08.686717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:08.686694    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"
Oct 30 23:09:08.686717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:08.686722    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c170405c1a513492fa344b1b84526e86305603c20242d5c3772db92b8da1b6e9"
Oct 30 23:09:10.816707 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:09:10.816659    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:09:26.779271 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.779205    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"]
Oct 30 23:09:26.779902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.779821    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6c7badff-2f63-4a26-ba3e-87ad50fd4d18" containerName="revoke-keys"
Oct 30 23:09:26.779902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.779842    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7badff-2f63-4a26-ba3e-87ad50fd4d18" containerName="revoke-keys"
Oct 30 23:09:26.780043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.780014    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7badff-2f63-4a26-ba3e-87ad50fd4d18" containerName="revoke-keys"
Oct 30 23:09:26.784779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.784757    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.801626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.801596    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:09:26.812147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.812084    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:09:26.812379 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.812245    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:09:26.817092 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.817062    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"]
Oct 30 23:09:26.856641 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.856603    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.856829 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.856649    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-kube-api-access-qk42v\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.856829 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.856676    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.901100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.901069    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:09:26.957771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.957728    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.957965 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.957789    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-kube-api-access-qk42v\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.957965 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.957827    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.958674 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.958593    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.960418 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.960394    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:26.975673 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:26.975639    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-kube-api-access-qk42v\") pod \"maas-api-revoke-sub-a4aef7d7-qqqsj\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:27.098440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:27.098352    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:27.400704 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:27.400673    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"]
Oct 30 23:09:27.403605 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:09:27.403576    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbbbe7a_82a7_49bc_9995_3e7bdb083c39.slice/crio-d69ee46a33e8d3b61d3d82fe474642fc5f313d7605501a812a51b9bbcbf3997c WatchSource:0}: Error finding container d69ee46a33e8d3b61d3d82fe474642fc5f313d7605501a812a51b9bbcbf3997c: Status 404 returned error can't find the container with id d69ee46a33e8d3b61d3d82fe474642fc5f313d7605501a812a51b9bbcbf3997c
Oct 30 23:09:27.864031 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:27.863990    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj" event={"ID":"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39","Type":"ContainerStarted","Data":"8011aeefe20d7ef3bd5ea283d541b0bb101d6a1a07f77facf0caae3c7eb0e35c"}
Oct 30 23:09:27.864484 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:27.864054    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj" event={"ID":"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39","Type":"ContainerStarted","Data":"d69ee46a33e8d3b61d3d82fe474642fc5f313d7605501a812a51b9bbcbf3997c"}
Oct 30 23:09:27.902753 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:27.902688    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj" podStartSLOduration=1.9026679610000001 podStartE2EDuration="1.902667961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:09:27.900021888 +0000 UTC m=+3751.938960775" watchObservedRunningTime="2026-10-30 23:09:27.902667961 +0000 UTC m=+3751.941606848"
Oct 30 23:09:28.872432 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:28.872393    4581 generic.go:356] "Generic (PLEG): container finished" podID="0cbbbe7a-82a7-49bc-9995-3e7bdb083c39" containerID="8011aeefe20d7ef3bd5ea283d541b0bb101d6a1a07f77facf0caae3c7eb0e35c" exitCode=0
Oct 30 23:09:28.872930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:28.872487    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj" event={"ID":"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39","Type":"ContainerDied","Data":"8011aeefe20d7ef3bd5ea283d541b0bb101d6a1a07f77facf0caae3c7eb0e35c"}
Oct 30 23:09:30.019170 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.019145    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:30.190768 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.190675    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-xks-ca\") pod \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") "
Oct 30 23:09:30.190950 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.190831    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-kube-api-access-qk42v\") pod \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") "
Oct 30 23:09:30.190950 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.190931    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-openshift-ca\") pod \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\" (UID: \"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39\") "
Oct 30 23:09:30.191339 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.191312    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-openshift-ca" pod "0cbbbe7a-82a7-49bc-9995-3e7bdb083c39" (UID: "0cbbbe7a-82a7-49bc-9995-3e7bdb083c39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:09:30.193001 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.192977    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-xks-ca" pod "0cbbbe7a-82a7-49bc-9995-3e7bdb083c39" (UID: "0cbbbe7a-82a7-49bc-9995-3e7bdb083c39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:09:30.193140 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.193119    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-kube-api-access-qk42v" pod "0cbbbe7a-82a7-49bc-9995-3e7bdb083c39" (UID: "0cbbbe7a-82a7-49bc-9995-3e7bdb083c39"). InnerVolumeSpecName "kube-api-access-qk42v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:09:30.291966 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.291927    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-kube-api-access-qk42v\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:30.291966 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.291958    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:30.291966 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.291968    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:30.887642 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.887611    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"
Oct 30 23:09:30.887837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.887645    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj" event={"ID":"0cbbbe7a-82a7-49bc-9995-3e7bdb083c39","Type":"ContainerDied","Data":"d69ee46a33e8d3b61d3d82fe474642fc5f313d7605501a812a51b9bbcbf3997c"}
Oct 30 23:09:30.887837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:30.887700    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d69ee46a33e8d3b61d3d82fe474642fc5f313d7605501a812a51b9bbcbf3997c"
Oct 30 23:09:34.030440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:34.030397    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"]
Oct 30 23:09:34.040421 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:34.040391    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e8faf-qqgrn"]
Oct 30 23:09:34.043568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:34.043538    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"]
Oct 30 23:09:34.046187 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:34.046159    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21908a-w2fxm"]
Oct 30 23:09:34.097880 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:34.097836    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c35483-4333-4938-9d5f-ed2e7330b588" path="/var/lib/kubelet/pods/88c35483-4333-4938-9d5f-ed2e7330b588/volumes"
Oct 30 23:09:34.098554 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:34.098531    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6495fe-1800-42a5-9b56-04c4b5a74745" path="/var/lib/kubelet/pods/cb6495fe-1800-42a5-9b56-04c4b5a74745/volumes"
Oct 30 23:09:40.816292 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:09:40.816252    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:09:46.956289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:46.956249    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"]
Oct 30 23:09:46.956796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:46.956706    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0cbbbe7a-82a7-49bc-9995-3e7bdb083c39" containerName="revoke-keys"
Oct 30 23:09:46.956796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:46.956721    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbbbe7a-82a7-49bc-9995-3e7bdb083c39" containerName="revoke-keys"
Oct 30 23:09:46.956796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:46.956794    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbbbe7a-82a7-49bc-9995-3e7bdb083c39" containerName="revoke-keys"
Oct 30 23:09:46.971792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:46.971758    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:46.979459 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:46.979419    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"]
Oct 30 23:09:46.985837 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:46.985806    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:09:46.986034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:46.985852    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:09:47.032984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.032946    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdnq\" (UniqueName: \"kubernetes.io/projected/8c6d2859-c74f-4b86-ab28-ac7412a412ed-kube-api-access-wmdnq\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.032984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.032985    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.033238 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.033088    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.037646 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.037623    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:09:47.038573 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.038554    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:09:47.133621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.133588    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdnq\" (UniqueName: \"kubernetes.io/projected/8c6d2859-c74f-4b86-ab28-ac7412a412ed-kube-api-access-wmdnq\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.133621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.133634    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.133902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.133705    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.134447 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.134425    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.136280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.136259    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.151917 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.151882    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdnq\" (UniqueName: \"kubernetes.io/projected/8c6d2859-c74f-4b86-ab28-ac7412a412ed-kube-api-access-wmdnq\") pod \"maas-api-revoke-sub-79805ca9-6dptq\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.291612 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.291509    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:47.594553 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:47.594523    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"]
Oct 30 23:09:47.597509 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:09:47.597481    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6d2859_c74f_4b86_ab28_ac7412a412ed.slice/crio-e4408132b6c7d26de072d8b3009f059023bf317a4e4aa9cd0c6a2a9255b0f497 WatchSource:0}: Error finding container e4408132b6c7d26de072d8b3009f059023bf317a4e4aa9cd0c6a2a9255b0f497: Status 404 returned error can't find the container with id e4408132b6c7d26de072d8b3009f059023bf317a4e4aa9cd0c6a2a9255b0f497
Oct 30 23:09:48.030084 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.030043    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"]
Oct 30 23:09:48.034257 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.034199    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"]
Oct 30 23:09:48.037373 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.037332    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq" event={"ID":"8c6d2859-c74f-4b86-ab28-ac7412a412ed","Type":"ContainerStarted","Data":"c2a4731f550382bac4482efdc1038d289c93b6b293a9260aa038f838ab6fd45e"}
Oct 30 23:09:48.037531 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.037389    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq" event={"ID":"8c6d2859-c74f-4b86-ab28-ac7412a412ed","Type":"ContainerStarted","Data":"e4408132b6c7d26de072d8b3009f059023bf317a4e4aa9cd0c6a2a9255b0f497"}
Oct 30 23:09:48.040447 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.040418    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d66fac1-5hwzg"]
Oct 30 23:09:48.044161 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.044132    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"]
Oct 30 23:09:48.046865 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.046839    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16b74c99-p6mcl"]
Oct 30 23:09:48.049382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.049361    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70972bb6-lqgzc"]
Oct 30 23:09:48.072487 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.072433    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq" podStartSLOduration=2.072416719 podStartE2EDuration="2.072416719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:09:48.071410933 +0000 UTC m=+3772.110349818" watchObservedRunningTime="2026-10-30 23:09:48.072416719 +0000 UTC m=+3772.111355605"
Oct 30 23:09:48.095533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.095506    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399109a1-199b-4b53-b2fb-a26a8b972eb2" path="/var/lib/kubelet/pods/399109a1-199b-4b53-b2fb-a26a8b972eb2/volumes"
Oct 30 23:09:48.095923 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.095911    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a7cd7b-ba2b-4086-aa42-b1db41b0165c" path="/var/lib/kubelet/pods/c0a7cd7b-ba2b-4086-aa42-b1db41b0165c/volumes"
Oct 30 23:09:48.096293 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:48.096279    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf4f622-4393-4170-a79e-ea0a1f0c0c93" path="/var/lib/kubelet/pods/faf4f622-4393-4170-a79e-ea0a1f0c0c93/volumes"
Oct 30 23:09:49.045787 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:49.045751    4581 generic.go:356] "Generic (PLEG): container finished" podID="8c6d2859-c74f-4b86-ab28-ac7412a412ed" containerID="c2a4731f550382bac4482efdc1038d289c93b6b293a9260aa038f838ab6fd45e" exitCode=0
Oct 30 23:09:49.046202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:49.045840    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq" event={"ID":"8c6d2859-c74f-4b86-ab28-ac7412a412ed","Type":"ContainerDied","Data":"c2a4731f550382bac4482efdc1038d289c93b6b293a9260aa038f838ab6fd45e"}
Oct 30 23:09:50.209590 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.209556    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:50.258545 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.258513    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-xks-ca\") pod \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") "
Oct 30 23:09:50.260793 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.260755    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-xks-ca" pod "8c6d2859-c74f-4b86-ab28-ac7412a412ed" (UID: "8c6d2859-c74f-4b86-ab28-ac7412a412ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:09:50.359038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.359008    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6d2859-c74f-4b86-ab28-ac7412a412ed-kube-api-access-wmdnq\" (UniqueName: \"kubernetes.io/projected/8c6d2859-c74f-4b86-ab28-ac7412a412ed-kube-api-access-wmdnq\") pod \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") "
Oct 30 23:09:50.359038 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.359045    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-openshift-ca\") pod \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\" (UID: \"8c6d2859-c74f-4b86-ab28-ac7412a412ed\") "
Oct 30 23:09:50.359401 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.359381    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:50.359512 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.359495    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-openshift-ca" pod "8c6d2859-c74f-4b86-ab28-ac7412a412ed" (UID: "8c6d2859-c74f-4b86-ab28-ac7412a412ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:09:50.361285 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.361265    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6d2859-c74f-4b86-ab28-ac7412a412ed-kube-api-access-wmdnq" pod "8c6d2859-c74f-4b86-ab28-ac7412a412ed" (UID: "8c6d2859-c74f-4b86-ab28-ac7412a412ed"). InnerVolumeSpecName "kube-api-access-wmdnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:09:50.460359 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.460320    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmdnq\" (UniqueName: \"kubernetes.io/projected/8c6d2859-c74f-4b86-ab28-ac7412a412ed-kube-api-access-wmdnq\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:50.460359 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:50.460349    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d2859-c74f-4b86-ab28-ac7412a412ed-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:09:51.070059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:51.070028    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"
Oct 30 23:09:51.070280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:51.070073    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq" event={"ID":"8c6d2859-c74f-4b86-ab28-ac7412a412ed","Type":"ContainerDied","Data":"e4408132b6c7d26de072d8b3009f059023bf317a4e4aa9cd0c6a2a9255b0f497"}
Oct 30 23:09:51.070280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:09:51.070122    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4408132b6c7d26de072d8b3009f059023bf317a4e4aa9cd0c6a2a9255b0f497"
Oct 30 23:10:03.542244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:03.541431    4581 scope.go:122] "RemoveContainer" containerID="12e70737ba9ee7187afa49686441f2927d9354feaa2cf83be85ef3d3e721dcd1"
Oct 30 23:10:03.581245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:03.578581    4581 scope.go:122] "RemoveContainer" containerID="7af6eb44c991d7b6d25b2faf8750da6f30ca26a3677580663a9f23233eaa634c"
Oct 30 23:10:03.645096 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:03.645067    4581 scope.go:122] "RemoveContainer" containerID="47f9cb25a32d13ec53238c8b92e4b4b9f594b12e0b48a97f9f2e96b8ab694d70"
Oct 30 23:10:03.696569 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:03.696522    4581 scope.go:122] "RemoveContainer" containerID="c0d0a646409845cdedec3ca777efa257404a599f0ac4e619638663cdd71de317"
Oct 30 23:10:03.740509 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:03.740485    4581 scope.go:122] "RemoveContainer" containerID="577cff746e4e4ee11b55508107762694a1cc834fd972e2c25462f4425e2175c7"
Oct 30 23:10:03.823312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:03.823281    4581 scope.go:122] "RemoveContainer" containerID="8047ec68bdc841786f7c9bfd1b647287b891054f729e762717ed9b0bfb02a53f"
Oct 30 23:10:03.900259 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:03.897420    4581 scope.go:122] "RemoveContainer" containerID="f78f9d0b1a8750a1594cb29e9d8519de0c2c1f9bff2e7b6e36ebff33433b0729"
Oct 30 23:10:03.956834 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:03.956675    4581 scope.go:122] "RemoveContainer" containerID="df6c89da0061c5051b0aa7ac4c593d75fb3dbee5c6d26699485a2835f1f1fdc6"
Oct 30 23:10:04.030968 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:04.030817    4581 scope.go:122] "RemoveContainer" containerID="5dc1947500be6ee1d5e4b607f0ec78ff0695a2ee900d5611b42afd24e2cdfb81"
Oct 30 23:10:04.079985 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:04.078304    4581 scope.go:122] "RemoveContainer" containerID="0699dc731fbb12710f978fef08f906217bb0e274a59f5d38436dcbde069bb72d"
Oct 30 23:10:04.116439 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:04.115956    4581 scope.go:122] "RemoveContainer" containerID="f7b28e1231f7a1e1cc548fa75accd5dc27ae35fe6f1ef330068414ce6fc57029"
Oct 30 23:10:06.985978 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:06.985943    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"]
Oct 30 23:10:06.986639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:06.986379    4581 cpu_manager.go:414] "containerMap: removing container" podUID="8c6d2859-c74f-4b86-ab28-ac7412a412ed" containerName="revoke-keys"
Oct 30 23:10:06.986639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:06.986394    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6d2859-c74f-4b86-ab28-ac7412a412ed" containerName="revoke-keys"
Oct 30 23:10:06.986639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:06.986487    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6d2859-c74f-4b86-ab28-ac7412a412ed" containerName="revoke-keys"
Oct 30 23:10:06.991939 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:06.991913    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.006283 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.006248    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:10:07.016308 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.016277    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:10:07.016550 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.016300    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:10:07.020624 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.020598    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:10:07.024067 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.024034    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"]
Oct 30 23:10:07.081978 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.081933    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.082185 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.082060    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98q88\" (UniqueName: \"kubernetes.io/projected/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-kube-api-access-98q88\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.082288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.082176    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.183160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.183123    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98q88\" (UniqueName: \"kubernetes.io/projected/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-kube-api-access-98q88\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.183412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.183246    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.183412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.183349    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.184163 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.184141    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.185869 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.185847    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.201600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.201567    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98q88\" (UniqueName: \"kubernetes.io/projected/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-kube-api-access-98q88\") pod \"maas-api-revoke-sub-0b7e8edb-gnbtm\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.305675 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.305584    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:07.609704 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:07.609671    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"]
Oct 30 23:10:07.612009 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:10:07.611983    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2d3b58_ee5b_404d_a4f8_894cbcde91f7.slice/crio-61c36e268763f91c2aeca9234a34f20437b83cefab14f849325c8b916ee08676 WatchSource:0}: Error finding container 61c36e268763f91c2aeca9234a34f20437b83cefab14f849325c8b916ee08676: Status 404 returned error can't find the container with id 61c36e268763f91c2aeca9234a34f20437b83cefab14f849325c8b916ee08676
Oct 30 23:10:08.255547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:08.255449    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm" event={"ID":"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7","Type":"ContainerStarted","Data":"28d2d83f416fab766e8449241f8619205ea22de85a201d31410dea221cfe8a0b"}
Oct 30 23:10:08.255547 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:08.255516    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm" event={"ID":"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7","Type":"ContainerStarted","Data":"61c36e268763f91c2aeca9234a34f20437b83cefab14f849325c8b916ee08676"}
Oct 30 23:10:08.290891 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:08.290840    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm" podStartSLOduration=2.290826741 podStartE2EDuration="2.290826741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:10:08.290082761 +0000 UTC m=+3792.329021646" watchObservedRunningTime="2026-10-30 23:10:08.290826741 +0000 UTC m=+3792.329765628"
Oct 30 23:10:09.263451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:09.263415    4581 generic.go:356] "Generic (PLEG): container finished" podID="1f2d3b58-ee5b-404d-a4f8-894cbcde91f7" containerID="28d2d83f416fab766e8449241f8619205ea22de85a201d31410dea221cfe8a0b" exitCode=0
Oct 30 23:10:09.263854 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:09.263468    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm" event={"ID":"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7","Type":"ContainerDied","Data":"28d2d83f416fab766e8449241f8619205ea22de85a201d31410dea221cfe8a0b"}
Oct 30 23:10:10.415966 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.415938    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:10.523320 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.523194    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-openshift-ca\") pod \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") "
Oct 30 23:10:10.523512 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.523350    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-kube-api-access-98q88\" (UniqueName: \"kubernetes.io/projected/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-kube-api-access-98q88\") pod \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") "
Oct 30 23:10:10.523512 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.523465    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-xks-ca\") pod \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\" (UID: \"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7\") "
Oct 30 23:10:10.523655 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.523601    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-openshift-ca" pod "1f2d3b58-ee5b-404d-a4f8-894cbcde91f7" (UID: "1f2d3b58-ee5b-404d-a4f8-894cbcde91f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:10:10.523799 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.523782    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:10.525675 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.525645    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-xks-ca" pod "1f2d3b58-ee5b-404d-a4f8-894cbcde91f7" (UID: "1f2d3b58-ee5b-404d-a4f8-894cbcde91f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:10:10.525815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.525672    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-kube-api-access-98q88" pod "1f2d3b58-ee5b-404d-a4f8-894cbcde91f7" (UID: "1f2d3b58-ee5b-404d-a4f8-894cbcde91f7"). InnerVolumeSpecName "kube-api-access-98q88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:10:10.625009 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.624964    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:10.625009 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:10.625002    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98q88\" (UniqueName: \"kubernetes.io/projected/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7-kube-api-access-98q88\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:10.816341 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:10:10.816254    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:10:11.278043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:11.278010    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"
Oct 30 23:10:11.278270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:11.278009    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm" event={"ID":"1f2d3b58-ee5b-404d-a4f8-894cbcde91f7","Type":"ContainerDied","Data":"61c36e268763f91c2aeca9234a34f20437b83cefab14f849325c8b916ee08676"}
Oct 30 23:10:11.278270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:11.278145    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61c36e268763f91c2aeca9234a34f20437b83cefab14f849325c8b916ee08676"
Oct 30 23:10:17.789773 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.789741    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"]
Oct 30 23:10:17.790190 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.790173    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1f2d3b58-ee5b-404d-a4f8-894cbcde91f7" containerName="revoke-keys"
Oct 30 23:10:17.790276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.790193    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2d3b58-ee5b-404d-a4f8-894cbcde91f7" containerName="revoke-keys"
Oct 30 23:10:17.790319 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.790305    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2d3b58-ee5b-404d-a4f8-894cbcde91f7" containerName="revoke-keys"
Oct 30 23:10:17.793700 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.793679    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:17.806560 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.806537    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:10:17.816671 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.816642    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:10:17.816872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.816642    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:10:17.816955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.816696    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:10:17.820736 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.820713    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"]
Oct 30 23:10:17.894658 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.894625    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:17.894864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.894767    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc82b\" (UniqueName: \"kubernetes.io/projected/316f9d3e-18c8-4152-a3c8-82b4e7b96564-kube-api-access-tc82b\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:17.894864 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.894813    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:17.995621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.995577    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc82b\" (UniqueName: \"kubernetes.io/projected/316f9d3e-18c8-4152-a3c8-82b4e7b96564-kube-api-access-tc82b\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:17.995813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.995635    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:17.995813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.995743    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:17.996481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.996449    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:17.998495 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:17.998462    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:18.013716 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:18.013682    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc82b\" (UniqueName: \"kubernetes.io/projected/316f9d3e-18c8-4152-a3c8-82b4e7b96564-kube-api-access-tc82b\") pod \"maas-api-revoke-sub-3d4156f1-b7dmw\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:18.108621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:18.108591    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:18.414151 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:18.414059    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"]
Oct 30 23:10:18.417958 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:10:18.417928    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316f9d3e_18c8_4152_a3c8_82b4e7b96564.slice/crio-ac29a6825093c45e7be781d1827852459b9be8e1592db415a5b6cd1cff713708 WatchSource:0}: Error finding container ac29a6825093c45e7be781d1827852459b9be8e1592db415a5b6cd1cff713708: Status 404 returned error can't find the container with id ac29a6825093c45e7be781d1827852459b9be8e1592db415a5b6cd1cff713708
Oct 30 23:10:19.333142 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:19.333103    4581 generic.go:356] "Generic (PLEG): container finished" podID="316f9d3e-18c8-4152-a3c8-82b4e7b96564" containerID="17d0efe7bafab190d60a7356d8ba4577e222ffa5d7b6863af93dbd199df195fa" exitCode=0
Oct 30 23:10:19.333602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:19.333190    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw" event={"ID":"316f9d3e-18c8-4152-a3c8-82b4e7b96564","Type":"ContainerDied","Data":"17d0efe7bafab190d60a7356d8ba4577e222ffa5d7b6863af93dbd199df195fa"}
Oct 30 23:10:19.333602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:19.333252    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw" event={"ID":"316f9d3e-18c8-4152-a3c8-82b4e7b96564","Type":"ContainerStarted","Data":"ac29a6825093c45e7be781d1827852459b9be8e1592db415a5b6cd1cff713708"}
Oct 30 23:10:20.484291 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.484259    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:20.631201 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.631158    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-openshift-ca\") pod \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") "
Oct 30 23:10:20.631201 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.631242    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316f9d3e-18c8-4152-a3c8-82b4e7b96564-kube-api-access-tc82b\" (UniqueName: \"kubernetes.io/projected/316f9d3e-18c8-4152-a3c8-82b4e7b96564-kube-api-access-tc82b\") pod \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") "
Oct 30 23:10:20.631523 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.631274    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-xks-ca\") pod \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\" (UID: \"316f9d3e-18c8-4152-a3c8-82b4e7b96564\") "
Oct 30 23:10:20.631644 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.631617    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-openshift-ca" pod "316f9d3e-18c8-4152-a3c8-82b4e7b96564" (UID: "316f9d3e-18c8-4152-a3c8-82b4e7b96564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:10:20.633466 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.633442    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-xks-ca" pod "316f9d3e-18c8-4152-a3c8-82b4e7b96564" (UID: "316f9d3e-18c8-4152-a3c8-82b4e7b96564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:10:20.633585 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.633547    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316f9d3e-18c8-4152-a3c8-82b4e7b96564-kube-api-access-tc82b" pod "316f9d3e-18c8-4152-a3c8-82b4e7b96564" (UID: "316f9d3e-18c8-4152-a3c8-82b4e7b96564"). InnerVolumeSpecName "kube-api-access-tc82b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:10:20.732183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.732140    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:20.732183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.732174    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc82b\" (UniqueName: \"kubernetes.io/projected/316f9d3e-18c8-4152-a3c8-82b4e7b96564-kube-api-access-tc82b\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:20.732183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:20.732184    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316f9d3e-18c8-4152-a3c8-82b4e7b96564-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:21.347563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:21.347527    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"
Oct 30 23:10:21.347784 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:21.347559    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw" event={"ID":"316f9d3e-18c8-4152-a3c8-82b4e7b96564","Type":"ContainerDied","Data":"ac29a6825093c45e7be781d1827852459b9be8e1592db415a5b6cd1cff713708"}
Oct 30 23:10:21.347784 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:21.347612    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac29a6825093c45e7be781d1827852459b9be8e1592db415a5b6cd1cff713708"
Oct 30 23:10:40.816486 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:10:40.816398    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:10:41.210420 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.210387    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"]
Oct 30 23:10:41.219965 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.219937    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b7dmw"]
Oct 30 23:10:41.249592 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.249552    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"]
Oct 30 23:10:41.250107 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.250087    4581 cpu_manager.go:414] "containerMap: removing container" podUID="316f9d3e-18c8-4152-a3c8-82b4e7b96564" containerName="revoke-keys"
Oct 30 23:10:41.250197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.250109    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316f9d3e-18c8-4152-a3c8-82b4e7b96564" containerName="revoke-keys"
Oct 30 23:10:41.250197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.250182    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="316f9d3e-18c8-4152-a3c8-82b4e7b96564" containerName="revoke-keys"
Oct 30 23:10:41.253449 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.253432    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.266902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.266874    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:10:41.266902 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.266881    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:10:41.267092 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.266925    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:10:41.267092 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.266985    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:10:41.273105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.273079    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"]
Oct 30 23:10:41.334621 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.334588    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.334830 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.334701    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnlj\" (UniqueName: \"kubernetes.io/projected/e54308b5-d0bf-4d3f-967a-0348c48c2bed-kube-api-access-bgnlj\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.334830 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.334747    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.435605 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.435564    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnlj\" (UniqueName: \"kubernetes.io/projected/e54308b5-d0bf-4d3f-967a-0348c48c2bed-kube-api-access-bgnlj\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.435605 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.435608    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.435887 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.435686    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.436305 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.436285    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.438327 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.438298    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.453173 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.453147    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnlj\" (UniqueName: \"kubernetes.io/projected/e54308b5-d0bf-4d3f-967a-0348c48c2bed-kube-api-access-bgnlj\") pod \"maas-api-revoke-sub-3d4156f1-dmd8q\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.567382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.567298    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:41.867124 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:41.867097    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"]
Oct 30 23:10:41.869694 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:10:41.869665    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54308b5_d0bf_4d3f_967a_0348c48c2bed.slice/crio-590b1d26cb06e3caa38d7b9df107988a2b63a3ac8de81359e268ef42ba059afa WatchSource:0}: Error finding container 590b1d26cb06e3caa38d7b9df107988a2b63a3ac8de81359e268ef42ba059afa: Status 404 returned error can't find the container with id 590b1d26cb06e3caa38d7b9df107988a2b63a3ac8de81359e268ef42ba059afa
Oct 30 23:10:42.094895 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:42.094863    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316f9d3e-18c8-4152-a3c8-82b4e7b96564" path="/var/lib/kubelet/pods/316f9d3e-18c8-4152-a3c8-82b4e7b96564/volumes"
Oct 30 23:10:42.614615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:42.614579    4581 generic.go:356] "Generic (PLEG): container finished" podID="e54308b5-d0bf-4d3f-967a-0348c48c2bed" containerID="c04cc7577efe4b4280bfda9514e22de2c6270bd1f3a69db73a342424b9e83862" exitCode=0
Oct 30 23:10:42.614817 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:42.614674    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q" event={"ID":"e54308b5-d0bf-4d3f-967a-0348c48c2bed","Type":"ContainerDied","Data":"c04cc7577efe4b4280bfda9514e22de2c6270bd1f3a69db73a342424b9e83862"}
Oct 30 23:10:42.614817 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:42.614773    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q" event={"ID":"e54308b5-d0bf-4d3f-967a-0348c48c2bed","Type":"ContainerStarted","Data":"590b1d26cb06e3caa38d7b9df107988a2b63a3ac8de81359e268ef42ba059afa"}
Oct 30 23:10:43.780787 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.780760    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:43.855110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.855071    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-xks-ca\") pod \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") "
Oct 30 23:10:43.855319 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.855140    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54308b5-d0bf-4d3f-967a-0348c48c2bed-kube-api-access-bgnlj\" (UniqueName: \"kubernetes.io/projected/e54308b5-d0bf-4d3f-967a-0348c48c2bed-kube-api-access-bgnlj\") pod \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") "
Oct 30 23:10:43.855319 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.855250    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-openshift-ca\") pod \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\" (UID: \"e54308b5-d0bf-4d3f-967a-0348c48c2bed\") "
Oct 30 23:10:43.855643 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.855617    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-openshift-ca" pod "e54308b5-d0bf-4d3f-967a-0348c48c2bed" (UID: "e54308b5-d0bf-4d3f-967a-0348c48c2bed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:10:43.857401 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.857371    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-xks-ca" pod "e54308b5-d0bf-4d3f-967a-0348c48c2bed" (UID: "e54308b5-d0bf-4d3f-967a-0348c48c2bed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:10:43.857538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.857513    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54308b5-d0bf-4d3f-967a-0348c48c2bed-kube-api-access-bgnlj" pod "e54308b5-d0bf-4d3f-967a-0348c48c2bed" (UID: "e54308b5-d0bf-4d3f-967a-0348c48c2bed"). InnerVolumeSpecName "kube-api-access-bgnlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:10:43.957053 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.956961    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:43.957053 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.956996    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnlj\" (UniqueName: \"kubernetes.io/projected/e54308b5-d0bf-4d3f-967a-0348c48c2bed-kube-api-access-bgnlj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:43.957053 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:43.957006    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54308b5-d0bf-4d3f-967a-0348c48c2bed-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:10:44.628981 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:44.628953    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"
Oct 30 23:10:44.628981 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:44.628963    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q" event={"ID":"e54308b5-d0bf-4d3f-967a-0348c48c2bed","Type":"ContainerDied","Data":"590b1d26cb06e3caa38d7b9df107988a2b63a3ac8de81359e268ef42ba059afa"}
Oct 30 23:10:44.629316 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:44.629005    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="590b1d26cb06e3caa38d7b9df107988a2b63a3ac8de81359e268ef42ba059afa"
Oct 30 23:10:46.032177 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:46.032136    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"]
Oct 30 23:10:46.041392 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:46.041362    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b7132f-shjq8"]
Oct 30 23:10:46.094723 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:10:46.094685    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8bbd476-8aef-4abe-924c-d6e4547612fe" path="/var/lib/kubelet/pods/d8bbd476-8aef-4abe-924c-d6e4547612fe/volumes"
Oct 30 23:11:04.260246 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.260040    4581 scope.go:122] "RemoveContainer" containerID="5ad59a776e304c0d1c896d7baca9965ab5c828d58b2d1751712b4d653db6c518"
Oct 30 23:11:04.303579 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.297467    4581 scope.go:122] "RemoveContainer" containerID="28a2606219818752083164374df1c3bc118e741b121571844d35ea3a71160305"
Oct 30 23:11:04.338833 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.338805    4581 scope.go:122] "RemoveContainer" containerID="e10eb43a95efaf639bd0f1d7637922eef09e7c137414e7d61bfb00593356daed"
Oct 30 23:11:04.401673 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.401640    4581 scope.go:122] "RemoveContainer" containerID="b5e3d09c52d4cd4258c454abf98218ce866c3c491f711feb41d1a41dd650814c"
Oct 30 23:11:04.478056 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.477919    4581 scope.go:122] "RemoveContainer" containerID="06bd41e31e1caf811ad32efeb83e78dac2587f1dd62678075e997d496feb110f"
Oct 30 23:11:04.539404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.539263    4581 scope.go:122] "RemoveContainer" containerID="7ec18a0b3f673a0aa546860afd22b078415f3cc77026fa9271f662431676996d"
Oct 30 23:11:04.572824 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.572697    4581 scope.go:122] "RemoveContainer" containerID="38dc3567ea9a92df8856e6763bcabf3d5955de0dacd6b0e0629cbee6b1a20e60"
Oct 30 23:11:04.621785 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.621383    4581 scope.go:122] "RemoveContainer" containerID="b603428e49d4020febacc5e18e2edf1ff8131dc64b8e0524cdf257ff1043ff7c"
Oct 30 23:11:04.717242 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.712409    4581 scope.go:122] "RemoveContainer" containerID="2c9954c219d1ffc727fa4678c7a3e7351beda9ae7a16be604da094b06bbe25c7"
Oct 30 23:11:04.825251 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.819610    4581 scope.go:122] "RemoveContainer" containerID="0f41a172454832f90edc71532b5178a0d048d4e90ba1e8bdca98ba907ca92e99"
Oct 30 23:11:04.923427 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:04.917091    4581 scope.go:122] "RemoveContainer" containerID="439fb731ee190db9be3c39982dc0ce9ebd3bb6a6b315fed8243f2e05c6d804ab"
Oct 30 23:11:08.748810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.748777    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"]
Oct 30 23:11:08.749230 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.749199    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e54308b5-d0bf-4d3f-967a-0348c48c2bed" containerName="revoke-keys"
Oct 30 23:11:08.749273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.749212    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54308b5-d0bf-4d3f-967a-0348c48c2bed" containerName="revoke-keys"
Oct 30 23:11:08.749319 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.749309    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54308b5-d0bf-4d3f-967a-0348c48c2bed" containerName="revoke-keys"
Oct 30 23:11:08.754698 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.754676    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.779538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.779497    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:11:08.779538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.779529    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:11:08.779796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.779508    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:11:08.779796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.779685    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:11:08.782994 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.782963    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"]
Oct 30 23:11:08.832462 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.832426    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.832679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.832467    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-448wz\" (UniqueName: \"kubernetes.io/projected/c23a0489-72af-4919-808c-17cb7f818cb0-kube-api-access-448wz\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.832679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.832581    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.934159 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.934121    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.934378 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.934166    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-448wz\" (UniqueName: \"kubernetes.io/projected/c23a0489-72af-4919-808c-17cb7f818cb0-kube-api-access-448wz\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.934378 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.934242    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.934944 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.934916    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.936737 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.936711    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:08.951879 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:08.951851    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-448wz\" (UniqueName: \"kubernetes.io/projected/c23a0489-72af-4919-808c-17cb7f818cb0-kube-api-access-448wz\") pod \"maas-api-revoke-sub-e2fc9c5b-gr6mf\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:09.067280 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:09.067183    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:09.370373 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:09.370348    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"]
Oct 30 23:11:09.373134 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:11:09.373106    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23a0489_72af_4919_808c_17cb7f818cb0.slice/crio-a8d8167633c876662fcc3611282c4506adb32638051514a9efc6a90105d357b6 WatchSource:0}: Error finding container a8d8167633c876662fcc3611282c4506adb32638051514a9efc6a90105d357b6: Status 404 returned error can't find the container with id a8d8167633c876662fcc3611282c4506adb32638051514a9efc6a90105d357b6
Oct 30 23:11:09.918524 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:09.918417    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf" event={"ID":"c23a0489-72af-4919-808c-17cb7f818cb0","Type":"ContainerStarted","Data":"de6186773be9edaaa722cb9599298b69d3c7eea7f4e7ef00395da233ab42135e"}
Oct 30 23:11:09.918524 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:09.918487    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf" event={"ID":"c23a0489-72af-4919-808c-17cb7f818cb0","Type":"ContainerStarted","Data":"a8d8167633c876662fcc3611282c4506adb32638051514a9efc6a90105d357b6"}
Oct 30 23:11:09.954020 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:09.953966    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf" podStartSLOduration=1.9539467940000002 podStartE2EDuration="1.953946794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:09.951964029 +0000 UTC m=+3853.990902915" watchObservedRunningTime="2026-10-30 23:11:09.953946794 +0000 UTC m=+3853.992885688"
Oct 30 23:11:10.816209 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:11:10.816170    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:11:10.925788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:10.925752    4581 generic.go:356] "Generic (PLEG): container finished" podID="c23a0489-72af-4919-808c-17cb7f818cb0" containerID="de6186773be9edaaa722cb9599298b69d3c7eea7f4e7ef00395da233ab42135e" exitCode=0
Oct 30 23:11:10.926247 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:10.925788    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf" event={"ID":"c23a0489-72af-4919-808c-17cb7f818cb0","Type":"ContainerDied","Data":"de6186773be9edaaa722cb9599298b69d3c7eea7f4e7ef00395da233ab42135e"}
Oct 30 23:11:12.063307 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.063284    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:12.166842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.166809    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23a0489-72af-4919-808c-17cb7f818cb0-kube-api-access-448wz\" (UniqueName: \"kubernetes.io/projected/c23a0489-72af-4919-808c-17cb7f818cb0-kube-api-access-448wz\") pod \"c23a0489-72af-4919-808c-17cb7f818cb0\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") "
Oct 30 23:11:12.167031 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.166851    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-openshift-ca\") pod \"c23a0489-72af-4919-808c-17cb7f818cb0\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") "
Oct 30 23:11:12.167031 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.166898    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-xks-ca\") pod \"c23a0489-72af-4919-808c-17cb7f818cb0\" (UID: \"c23a0489-72af-4919-808c-17cb7f818cb0\") "
Oct 30 23:11:12.167301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.167263    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-openshift-ca" pod "c23a0489-72af-4919-808c-17cb7f818cb0" (UID: "c23a0489-72af-4919-808c-17cb7f818cb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:11:12.169105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.169072    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-xks-ca" pod "c23a0489-72af-4919-808c-17cb7f818cb0" (UID: "c23a0489-72af-4919-808c-17cb7f818cb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:11:12.169182 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.169142    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23a0489-72af-4919-808c-17cb7f818cb0-kube-api-access-448wz" pod "c23a0489-72af-4919-808c-17cb7f818cb0" (UID: "c23a0489-72af-4919-808c-17cb7f818cb0"). InnerVolumeSpecName "kube-api-access-448wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:11:12.268110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.268022    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-448wz\" (UniqueName: \"kubernetes.io/projected/c23a0489-72af-4919-808c-17cb7f818cb0-kube-api-access-448wz\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:12.268110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.268055    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:12.268110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.268068    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a0489-72af-4919-808c-17cb7f818cb0-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:12.939542 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.939516    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"
Oct 30 23:11:12.939733 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.939516    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf" event={"ID":"c23a0489-72af-4919-808c-17cb7f818cb0","Type":"ContainerDied","Data":"a8d8167633c876662fcc3611282c4506adb32638051514a9efc6a90105d357b6"}
Oct 30 23:11:12.939733 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:12.939644    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8d8167633c876662fcc3611282c4506adb32638051514a9efc6a90105d357b6"
Oct 30 23:11:19.088681 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.088640    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"]
Oct 30 23:11:19.089247 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.089232    4581 cpu_manager.go:414] "containerMap: removing container" podUID="c23a0489-72af-4919-808c-17cb7f818cb0" containerName="revoke-keys"
Oct 30 23:11:19.089304 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.089250    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23a0489-72af-4919-808c-17cb7f818cb0" containerName="revoke-keys"
Oct 30 23:11:19.089399 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.089388    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23a0489-72af-4919-808c-17cb7f818cb0" containerName="revoke-keys"
Oct 30 23:11:19.093416 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.093386    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.106404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.106378    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:11:19.115112 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.115081    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:11:19.115283 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.115130    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:11:19.115283 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.115162    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:11:19.120897 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.120869    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"]
Oct 30 23:11:19.131565 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.131524    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.131771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.131579    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.131771 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.131749    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q898p\" (UniqueName: \"kubernetes.io/projected/46ff7db6-fea4-4170-9fae-b6d963faeff3-kube-api-access-q898p\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.232459 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.232423    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q898p\" (UniqueName: \"kubernetes.io/projected/46ff7db6-fea4-4170-9fae-b6d963faeff3-kube-api-access-q898p\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.232664 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.232500    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.232664 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.232521    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.233210 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.233183    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.234941 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.234918    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.249438 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.249413    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q898p\" (UniqueName: \"kubernetes.io/projected/46ff7db6-fea4-4170-9fae-b6d963faeff3-kube-api-access-q898p\") pod \"maas-api-revoke-sub-5a29efb3-6t9mr\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.408382 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.408345    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:19.708984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.708903    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"]
Oct 30 23:11:19.712518 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:11:19.712490    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ff7db6_fea4_4170_9fae_b6d963faeff3.slice/crio-aa5e92b0838b8635e070f55c00fc78e7a5a9954204de29286a2a5df45684e5c1 WatchSource:0}: Error finding container aa5e92b0838b8635e070f55c00fc78e7a5a9954204de29286a2a5df45684e5c1: Status 404 returned error can't find the container with id aa5e92b0838b8635e070f55c00fc78e7a5a9954204de29286a2a5df45684e5c1
Oct 30 23:11:19.983314 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:19.983282    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr" event={"ID":"46ff7db6-fea4-4170-9fae-b6d963faeff3","Type":"ContainerStarted","Data":"aa5e92b0838b8635e070f55c00fc78e7a5a9954204de29286a2a5df45684e5c1"}
Oct 30 23:11:20.996531 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:20.996424    4581 generic.go:356] "Generic (PLEG): container finished" podID="46ff7db6-fea4-4170-9fae-b6d963faeff3" containerID="13c5c6dfa716e6ad64bf35da9ab9ff42587a0dbb8d5722c5e293d8dc8347739d" exitCode=0
Oct 30 23:11:20.997147 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:20.996504    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr" event={"ID":"46ff7db6-fea4-4170-9fae-b6d963faeff3","Type":"ContainerDied","Data":"13c5c6dfa716e6ad64bf35da9ab9ff42587a0dbb8d5722c5e293d8dc8347739d"}
Oct 30 23:11:22.992000 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:22.991656    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:23.040793 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.040756    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr" event={"ID":"46ff7db6-fea4-4170-9fae-b6d963faeff3","Type":"ContainerDied","Data":"aa5e92b0838b8635e070f55c00fc78e7a5a9954204de29286a2a5df45684e5c1"}
Oct 30 23:11:23.041059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.041041    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5e92b0838b8635e070f55c00fc78e7a5a9954204de29286a2a5df45684e5c1"
Oct 30 23:11:23.041258 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.041242    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"
Oct 30 23:11:23.166647 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.166532    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-xks-ca\") pod \"46ff7db6-fea4-4170-9fae-b6d963faeff3\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") "
Oct 30 23:11:23.166976 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.166939    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-openshift-ca\") pod \"46ff7db6-fea4-4170-9fae-b6d963faeff3\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") "
Oct 30 23:11:23.167173 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.167137    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ff7db6-fea4-4170-9fae-b6d963faeff3-kube-api-access-q898p\" (UniqueName: \"kubernetes.io/projected/46ff7db6-fea4-4170-9fae-b6d963faeff3-kube-api-access-q898p\") pod \"46ff7db6-fea4-4170-9fae-b6d963faeff3\" (UID: \"46ff7db6-fea4-4170-9fae-b6d963faeff3\") "
Oct 30 23:11:23.169062 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.169022    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-openshift-ca" pod "46ff7db6-fea4-4170-9fae-b6d963faeff3" (UID: "46ff7db6-fea4-4170-9fae-b6d963faeff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:11:23.186289 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.186204    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ff7db6-fea4-4170-9fae-b6d963faeff3-kube-api-access-q898p" pod "46ff7db6-fea4-4170-9fae-b6d963faeff3" (UID: "46ff7db6-fea4-4170-9fae-b6d963faeff3"). InnerVolumeSpecName "kube-api-access-q898p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:11:23.188669 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.187619    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-xks-ca" pod "46ff7db6-fea4-4170-9fae-b6d963faeff3" (UID: "46ff7db6-fea4-4170-9fae-b6d963faeff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:11:23.270061 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.268966    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q898p\" (UniqueName: \"kubernetes.io/projected/46ff7db6-fea4-4170-9fae-b6d963faeff3-kube-api-access-q898p\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:23.270061 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.269014    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:23.270061 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:23.269031    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff7db6-fea4-4170-9fae-b6d963faeff3-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:36.029112 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:36.029071    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"]
Oct 30 23:11:36.038208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:36.038180    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-dsnvx"]
Oct 30 23:11:36.098146 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:36.098108    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4203df-9064-48dd-8e87-dbd582e681b2" path="/var/lib/kubelet/pods/ac4203df-9064-48dd-8e87-dbd582e681b2/volumes"
Oct 30 23:11:38.330100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.330068    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"]
Oct 30 23:11:38.339596 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.339568    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmd8q"]
Oct 30 23:11:38.374528 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.374495    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"]
Oct 30 23:11:38.374951 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.374937    4581 cpu_manager.go:414] "containerMap: removing container" podUID="46ff7db6-fea4-4170-9fae-b6d963faeff3" containerName="revoke-keys"
Oct 30 23:11:38.375004 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.374953    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ff7db6-fea4-4170-9fae-b6d963faeff3" containerName="revoke-keys"
Oct 30 23:11:38.375045 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.375025    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ff7db6-fea4-4170-9fae-b6d963faeff3" containerName="revoke-keys"
Oct 30 23:11:38.378335 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.378314    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.390589 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.390560    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:11:38.390709 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.390618    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:11:38.390709 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.390642    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:11:38.390785 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.390718    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:11:38.403597 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.403567    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"]
Oct 30 23:11:38.416263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.413687    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.416263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.413822    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.416263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.413873    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwvj\" (UniqueName: \"kubernetes.io/projected/28236c6c-bca4-4fc7-bfee-4ab055ada51b-kube-api-access-qrwvj\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.514568 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.514532    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.514779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.514659    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.514779 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.514712    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwvj\" (UniqueName: \"kubernetes.io/projected/28236c6c-bca4-4fc7-bfee-4ab055ada51b-kube-api-access-qrwvj\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.515068 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.515048    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.517166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.517148    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.532950 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.532924    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwvj\" (UniqueName: \"kubernetes.io/projected/28236c6c-bca4-4fc7-bfee-4ab055ada51b-kube-api-access-qrwvj\") pod \"maas-api-revoke-sub-3d4156f1-l65r4\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.691593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.691559    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:38.992630 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:38.992545    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"]
Oct 30 23:11:39.000016 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:11:38.996338    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28236c6c_bca4_4fc7_bfee_4ab055ada51b.slice/crio-6f0df0b0e57cd2727b49826ff9305c9454cf3d3fc25f1dcc2adbe94857517149 WatchSource:0}: Error finding container 6f0df0b0e57cd2727b49826ff9305c9454cf3d3fc25f1dcc2adbe94857517149: Status 404 returned error can't find the container with id 6f0df0b0e57cd2727b49826ff9305c9454cf3d3fc25f1dcc2adbe94857517149
Oct 30 23:11:39.139977 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:39.139938    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4" event={"ID":"28236c6c-bca4-4fc7-bfee-4ab055ada51b","Type":"ContainerStarted","Data":"6f0df0b0e57cd2727b49826ff9305c9454cf3d3fc25f1dcc2adbe94857517149"}
Oct 30 23:11:40.095887 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:40.095803    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54308b5-d0bf-4d3f-967a-0348c48c2bed" path="/var/lib/kubelet/pods/e54308b5-d0bf-4d3f-967a-0348c48c2bed/volumes"
Oct 30 23:11:40.147635 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:40.147592    4581 generic.go:356] "Generic (PLEG): container finished" podID="28236c6c-bca4-4fc7-bfee-4ab055ada51b" containerID="7b6c545d7db400ba915686de9aaae5cc6663784cafc047a88b6a5f77b45949fb" exitCode=0
Oct 30 23:11:40.147814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:40.147685    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4" event={"ID":"28236c6c-bca4-4fc7-bfee-4ab055ada51b","Type":"ContainerDied","Data":"7b6c545d7db400ba915686de9aaae5cc6663784cafc047a88b6a5f77b45949fb"}
Oct 30 23:11:40.815733 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:11:40.815688    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:11:41.290843 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.290811    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:41.341065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.341030    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28236c6c-bca4-4fc7-bfee-4ab055ada51b-kube-api-access-qrwvj\" (UniqueName: \"kubernetes.io/projected/28236c6c-bca4-4fc7-bfee-4ab055ada51b-kube-api-access-qrwvj\") pod \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") "
Oct 30 23:11:41.341253 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.341161    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-openshift-ca\") pod \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") "
Oct 30 23:11:41.341317 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.341287    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-xks-ca\") pod \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\" (UID: \"28236c6c-bca4-4fc7-bfee-4ab055ada51b\") "
Oct 30 23:11:41.341628 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.341595    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-openshift-ca" pod "28236c6c-bca4-4fc7-bfee-4ab055ada51b" (UID: "28236c6c-bca4-4fc7-bfee-4ab055ada51b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:11:41.344158 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.344132    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-xks-ca" pod "28236c6c-bca4-4fc7-bfee-4ab055ada51b" (UID: "28236c6c-bca4-4fc7-bfee-4ab055ada51b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:11:41.344297 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.344230    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28236c6c-bca4-4fc7-bfee-4ab055ada51b-kube-api-access-qrwvj" pod "28236c6c-bca4-4fc7-bfee-4ab055ada51b" (UID: "28236c6c-bca4-4fc7-bfee-4ab055ada51b"). InnerVolumeSpecName "kube-api-access-qrwvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:11:41.442370 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.442279    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:41.442370 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.442308    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwvj\" (UniqueName: \"kubernetes.io/projected/28236c6c-bca4-4fc7-bfee-4ab055ada51b-kube-api-access-qrwvj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:41.442370 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:41.442320    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28236c6c-bca4-4fc7-bfee-4ab055ada51b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:11:42.161481 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:42.161452    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"
Oct 30 23:11:42.161670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:42.161443    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4" event={"ID":"28236c6c-bca4-4fc7-bfee-4ab055ada51b","Type":"ContainerDied","Data":"6f0df0b0e57cd2727b49826ff9305c9454cf3d3fc25f1dcc2adbe94857517149"}
Oct 30 23:11:42.161670 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:42.161590    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f0df0b0e57cd2727b49826ff9305c9454cf3d3fc25f1dcc2adbe94857517149"
Oct 30 23:11:57.598611 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:57.598580    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:11:57.608850 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:11:57.608826    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:12:05.298117 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:05.298075    4581 scope.go:122] "RemoveContainer" containerID="0b45c750da8a16af87952319b64473fa30440b843cf71d29fff5bfaa22d0a5ea"
Oct 30 23:12:05.308112 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:05.308090    4581 scope.go:122] "RemoveContainer" containerID="5b4f1d515dfd55a37fbea38365ccfd397befcc04e501a0cf69d04c08af4857d9"
Oct 30 23:12:05.318252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:05.318209    4581 scope.go:122] "RemoveContainer" containerID="aa912c3eaf0e0608d1f74aa734b684c754efa6f058126db8abf8f28ea1808d48"
Oct 30 23:12:10.815685 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:12:10.815606    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:12:34.037397 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:34.037357    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"]
Oct 30 23:12:34.046632 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:34.046606    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8dmv4"]
Oct 30 23:12:34.096878 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:34.096841    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02167675-af5c-478e-998b-1f0f496450ba" path="/var/lib/kubelet/pods/02167675-af5c-478e-998b-1f0f496450ba/volumes"
Oct 30 23:12:40.816280 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:12:40.816239    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:12:47.186363 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.186327    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"]
Oct 30 23:12:47.186868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.186755    4581 cpu_manager.go:414] "containerMap: removing container" podUID="28236c6c-bca4-4fc7-bfee-4ab055ada51b" containerName="revoke-keys"
Oct 30 23:12:47.186868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.186769    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28236c6c-bca4-4fc7-bfee-4ab055ada51b" containerName="revoke-keys"
Oct 30 23:12:47.186868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.186854    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="28236c6c-bca4-4fc7-bfee-4ab055ada51b" containerName="revoke-keys"
Oct 30 23:12:47.190263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.190248    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.202987 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.202962    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:12:47.211954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.211919    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:12:47.212123 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.211987    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:12:47.212123 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.212032    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:12:47.216005 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.215980    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"]
Oct 30 23:12:47.295201 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.295162    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkzmj\" (UniqueName: \"kubernetes.io/projected/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-kube-api-access-tkzmj\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.295452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.295250    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.295452 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.295344    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.396412 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.396374    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkzmj\" (UniqueName: \"kubernetes.io/projected/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-kube-api-access-tkzmj\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.396626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.396440    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.396626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.396599    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.397303 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.397276    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.399123 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.399102    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.414257 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.414204    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkzmj\" (UniqueName: \"kubernetes.io/projected/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-kube-api-access-tkzmj\") pod \"maas-api-revoke-sub-07a579ea-7s2br\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.503544 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.503448    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:47.807937 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:47.807906    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"]
Oct 30 23:12:47.810847 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:12:47.810816    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174f3dd5_eb83_4931_b1d4_3f0f140cccaf.slice/crio-217240f817751eb6c1a0721d9f5e39d5aa094092418b42680965533686eadc08 WatchSource:0}: Error finding container 217240f817751eb6c1a0721d9f5e39d5aa094092418b42680965533686eadc08: Status 404 returned error can't find the container with id 217240f817751eb6c1a0721d9f5e39d5aa094092418b42680965533686eadc08
Oct 30 23:12:48.583058 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:48.583026    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br" event={"ID":"174f3dd5-eb83-4931-b1d4-3f0f140cccaf","Type":"ContainerStarted","Data":"068522a9bb728515539791102816303625ddb2075afb20b8f1bbf78f9021df26"}
Oct 30 23:12:48.583583 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:48.583089    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br" event={"ID":"174f3dd5-eb83-4931-b1d4-3f0f140cccaf","Type":"ContainerStarted","Data":"217240f817751eb6c1a0721d9f5e39d5aa094092418b42680965533686eadc08"}
Oct 30 23:12:48.621923 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:48.621755    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br" podStartSLOduration=1.621736364 podStartE2EDuration="1.621736364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:48.619207123 +0000 UTC m=+3952.658146010" watchObservedRunningTime="2026-10-30 23:12:48.621736364 +0000 UTC m=+3952.660675251"
Oct 30 23:12:49.591005 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:49.590969    4581 generic.go:356] "Generic (PLEG): container finished" podID="174f3dd5-eb83-4931-b1d4-3f0f140cccaf" containerID="068522a9bb728515539791102816303625ddb2075afb20b8f1bbf78f9021df26" exitCode=0
Oct 30 23:12:49.591431 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:49.591055    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br" event={"ID":"174f3dd5-eb83-4931-b1d4-3f0f140cccaf","Type":"ContainerDied","Data":"068522a9bb728515539791102816303625ddb2075afb20b8f1bbf78f9021df26"}
Oct 30 23:12:50.736511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.736484    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:50.828736 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.828704    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-openshift-ca\") pod \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") "
Oct 30 23:12:50.828923 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.828763    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-kube-api-access-tkzmj\" (UniqueName: \"kubernetes.io/projected/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-kube-api-access-tkzmj\") pod \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") "
Oct 30 23:12:50.828923 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.828821    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-xks-ca\") pod \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\" (UID: \"174f3dd5-eb83-4931-b1d4-3f0f140cccaf\") "
Oct 30 23:12:50.829166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.829139    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-openshift-ca" pod "174f3dd5-eb83-4931-b1d4-3f0f140cccaf" (UID: "174f3dd5-eb83-4931-b1d4-3f0f140cccaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:12:50.831001 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.830976    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-xks-ca" pod "174f3dd5-eb83-4931-b1d4-3f0f140cccaf" (UID: "174f3dd5-eb83-4931-b1d4-3f0f140cccaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:12:50.831127 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.831046    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-kube-api-access-tkzmj" pod "174f3dd5-eb83-4931-b1d4-3f0f140cccaf" (UID: "174f3dd5-eb83-4931-b1d4-3f0f140cccaf"). InnerVolumeSpecName "kube-api-access-tkzmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:12:50.930244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.930177    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:12:50.930244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.930210    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkzmj\" (UniqueName: \"kubernetes.io/projected/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-kube-api-access-tkzmj\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:12:50.930244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:50.930253    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174f3dd5-eb83-4931-b1d4-3f0f140cccaf-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:12:51.605178 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:51.605152    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"
Oct 30 23:12:51.605393 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:51.605190    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br" event={"ID":"174f3dd5-eb83-4931-b1d4-3f0f140cccaf","Type":"ContainerDied","Data":"217240f817751eb6c1a0721d9f5e39d5aa094092418b42680965533686eadc08"}
Oct 30 23:12:51.605393 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:51.605260    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="217240f817751eb6c1a0721d9f5e39d5aa094092418b42680965533686eadc08"
Oct 30 23:12:58.503145 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.503091    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"]
Oct 30 23:12:58.503699 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.503680    4581 cpu_manager.go:414] "containerMap: removing container" podUID="174f3dd5-eb83-4931-b1d4-3f0f140cccaf" containerName="revoke-keys"
Oct 30 23:12:58.503751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.503701    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174f3dd5-eb83-4931-b1d4-3f0f140cccaf" containerName="revoke-keys"
Oct 30 23:12:58.503804 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.503795    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="174f3dd5-eb83-4931-b1d4-3f0f140cccaf" containerName="revoke-keys"
Oct 30 23:12:58.507120 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.507102    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.529543 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.528995    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:12:58.529543 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.529039    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:12:58.529543 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.529060    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:12:58.529543 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.529065    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:12:58.534988 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.534954    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"]
Oct 30 23:12:58.609235 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.609186    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.609443 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.609250    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8hn\" (UniqueName: \"kubernetes.io/projected/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-kube-api-access-6j8hn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.609443 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.609350    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.710386 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.710340    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.710619 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.710406    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8hn\" (UniqueName: \"kubernetes.io/projected/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-kube-api-access-6j8hn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.710619 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.710427    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.711208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.711184    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.713071 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.713039    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.728882 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.728854    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8hn\" (UniqueName: \"kubernetes.io/projected/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-kube-api-access-6j8hn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:58.821577 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:58.821481    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:12:59.037013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:59.036975    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"]
Oct 30 23:12:59.039811 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:59.039785    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rvhn8"]
Oct 30 23:12:59.131591 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:59.131559    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"]
Oct 30 23:12:59.134548 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:12:59.134508    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1ef2a6_4d02_4f8b_9e6b_cdc2c68b7cc1.slice/crio-66363cda17897e0633cd7c6fdcd5f7666742012aaf47d86e564dbbfabd3f444d WatchSource:0}: Error finding container 66363cda17897e0633cd7c6fdcd5f7666742012aaf47d86e564dbbfabd3f444d: Status 404 returned error can't find the container with id 66363cda17897e0633cd7c6fdcd5f7666742012aaf47d86e564dbbfabd3f444d
Oct 30 23:12:59.666424 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:59.666378    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr" event={"ID":"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1","Type":"ContainerStarted","Data":"b16619bb7c7ebb7c1693f4718ec5968e242f103c2c59979da252764e61a9a3d4"}
Oct 30 23:12:59.666932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:59.666448    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr" event={"ID":"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1","Type":"ContainerStarted","Data":"66363cda17897e0633cd7c6fdcd5f7666742012aaf47d86e564dbbfabd3f444d"}
Oct 30 23:12:59.711488 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:12:59.711363    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr" podStartSLOduration=1.711340015 podStartE2EDuration="1.711340015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:59.706375372 +0000 UTC m=+3963.745314249" watchObservedRunningTime="2026-10-30 23:12:59.711340015 +0000 UTC m=+3963.750278905"
Oct 30 23:13:00.095922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:00.095857    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984c5fcd-1d88-4850-bfb0-0239f65fa2af" path="/var/lib/kubelet/pods/984c5fcd-1d88-4850-bfb0-0239f65fa2af/volumes"
Oct 30 23:13:00.676090 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:00.676046    4581 generic.go:356] "Generic (PLEG): container finished" podID="7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1" containerID="b16619bb7c7ebb7c1693f4718ec5968e242f103c2c59979da252764e61a9a3d4" exitCode=0
Oct 30 23:13:00.676600 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:00.676124    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr" event={"ID":"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1","Type":"ContainerDied","Data":"b16619bb7c7ebb7c1693f4718ec5968e242f103c2c59979da252764e61a9a3d4"}
Oct 30 23:13:01.831127 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.831101    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:13:01.851959 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.851925    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-xks-ca\") pod \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") "
Oct 30 23:13:01.852145 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.851991    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-kube-api-access-6j8hn\" (UniqueName: \"kubernetes.io/projected/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-kube-api-access-6j8hn\") pod \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") "
Oct 30 23:13:01.852145 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.852067    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-openshift-ca\") pod \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\" (UID: \"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1\") "
Oct 30 23:13:01.852533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.852498    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-openshift-ca" pod "7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1" (UID: "7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:13:01.854240 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.854191    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-xks-ca" pod "7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1" (UID: "7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:13:01.854359 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.854274    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-kube-api-access-6j8hn" pod "7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1" (UID: "7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1"). InnerVolumeSpecName "kube-api-access-6j8hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:01.953064 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.952974    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:13:01.953064 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.953010    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:13:01.953064 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:01.953020    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j8hn\" (UniqueName: \"kubernetes.io/projected/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1-kube-api-access-6j8hn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:13:02.692043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:02.692009    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"
Oct 30 23:13:02.692043 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:02.692030    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr" event={"ID":"7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1","Type":"ContainerDied","Data":"66363cda17897e0633cd7c6fdcd5f7666742012aaf47d86e564dbbfabd3f444d"}
Oct 30 23:13:02.692304 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:02.692082    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66363cda17897e0633cd7c6fdcd5f7666742012aaf47d86e564dbbfabd3f444d"
Oct 30 23:13:05.355154 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:05.353298    4581 scope.go:122] "RemoveContainer" containerID="a868dd77781b973aa7d458eb8e387b1ef85a73e90ba078de40c95a6af3a7d49e"
Oct 30 23:13:10.050955 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:10.050911    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"]
Oct 30 23:13:10.059617 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:10.059585    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sr2xw"]
Oct 30 23:13:10.095032 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:10.094996    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a27700-0280-4b3e-8dda-4e8e457bfd8b" path="/var/lib/kubelet/pods/71a27700-0280-4b3e-8dda-4e8e457bfd8b/volumes"
Oct 30 23:13:10.816557 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:13:10.816518    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:13:11.866580 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:11.866522    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"]
Oct 30 23:13:11.875910 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:11.875870    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cad7e-7v4lr"]
Oct 30 23:13:12.095810 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:12.095777    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1" path="/var/lib/kubelet/pods/7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1/volumes"
Oct 30 23:13:20.328431 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.328393    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"]
Oct 30 23:13:20.337526 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.337497    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l65r4"]
Oct 30 23:13:20.364744 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.364711    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"]
Oct 30 23:13:20.365120 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.365108    4581 cpu_manager.go:414] "containerMap: removing container" podUID="7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1" containerName="revoke-keys"
Oct 30 23:13:20.365168 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.365122    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1" containerName="revoke-keys"
Oct 30 23:13:20.365207 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.365200    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1ef2a6-4d02-4f8b-9e6b-cdc2c68b7cc1" containerName="revoke-keys"
Oct 30 23:13:20.369746 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.369724    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.382103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.382076    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:13:20.382270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.382104    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:13:20.382270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.382135    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:13:20.382270 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.382173    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:13:20.390478 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.390449    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"]
Oct 30 23:13:20.437294 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.437254    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.437478 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.437408    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.437478 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.437459    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2gr\" (UniqueName: \"kubernetes.io/projected/ee9f4532-6b89-474d-be88-8395dfbdb971-kube-api-access-sx2gr\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.538419 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.538381    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.538601 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.538435    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2gr\" (UniqueName: \"kubernetes.io/projected/ee9f4532-6b89-474d-be88-8395dfbdb971-kube-api-access-sx2gr\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.538601 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.538502    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.539032 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.539011    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.541063 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.541041    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.557762 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.557735    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2gr\" (UniqueName: \"kubernetes.io/projected/ee9f4532-6b89-474d-be88-8395dfbdb971-kube-api-access-sx2gr\") pod \"maas-api-revoke-sub-3d4156f1-knh5d\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.682774 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.682738    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:20.983533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:20.983444    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"]
Oct 30 23:13:20.987468 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:13:20.987437    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9f4532_6b89_474d_be88_8395dfbdb971.slice/crio-cd8cac44ea93dd00ea01fe8955332fdc93694f7a7b9888dba42d082e6b9c89d1 WatchSource:0}: Error finding container cd8cac44ea93dd00ea01fe8955332fdc93694f7a7b9888dba42d082e6b9c89d1: Status 404 returned error can't find the container with id cd8cac44ea93dd00ea01fe8955332fdc93694f7a7b9888dba42d082e6b9c89d1
Oct 30 23:13:21.813632 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:21.813601    4581 generic.go:356] "Generic (PLEG): container finished" podID="ee9f4532-6b89-474d-be88-8395dfbdb971" containerID="f572d14934e64e6ec9b3c43f9a9babb1a5680fb4104cdf1d28c812f915c7a0fe" exitCode=0
Oct 30 23:13:21.814067 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:21.813683    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d" event={"ID":"ee9f4532-6b89-474d-be88-8395dfbdb971","Type":"ContainerDied","Data":"f572d14934e64e6ec9b3c43f9a9babb1a5680fb4104cdf1d28c812f915c7a0fe"}
Oct 30 23:13:21.814067 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:21.813761    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d" event={"ID":"ee9f4532-6b89-474d-be88-8395dfbdb971","Type":"ContainerStarted","Data":"cd8cac44ea93dd00ea01fe8955332fdc93694f7a7b9888dba42d082e6b9c89d1"}
Oct 30 23:13:22.095326 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:22.095243    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28236c6c-bca4-4fc7-bfee-4ab055ada51b" path="/var/lib/kubelet/pods/28236c6c-bca4-4fc7-bfee-4ab055ada51b/volumes"
Oct 30 23:13:22.954883 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:22.954857    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:23.066937 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.066902    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-openshift-ca\") pod \"ee9f4532-6b89-474d-be88-8395dfbdb971\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") "
Oct 30 23:13:23.067119 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.066945    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-xks-ca\") pod \"ee9f4532-6b89-474d-be88-8395dfbdb971\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") "
Oct 30 23:13:23.067119 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.067042    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9f4532-6b89-474d-be88-8395dfbdb971-kube-api-access-sx2gr\" (UniqueName: \"kubernetes.io/projected/ee9f4532-6b89-474d-be88-8395dfbdb971-kube-api-access-sx2gr\") pod \"ee9f4532-6b89-474d-be88-8395dfbdb971\" (UID: \"ee9f4532-6b89-474d-be88-8395dfbdb971\") "
Oct 30 23:13:23.067305 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.067281    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-openshift-ca" pod "ee9f4532-6b89-474d-be88-8395dfbdb971" (UID: "ee9f4532-6b89-474d-be88-8395dfbdb971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:13:23.067443 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.067417    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:13:23.069367 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.069345    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-xks-ca" pod "ee9f4532-6b89-474d-be88-8395dfbdb971" (UID: "ee9f4532-6b89-474d-be88-8395dfbdb971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:13:23.069473 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.069406    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9f4532-6b89-474d-be88-8395dfbdb971-kube-api-access-sx2gr" pod "ee9f4532-6b89-474d-be88-8395dfbdb971" (UID: "ee9f4532-6b89-474d-be88-8395dfbdb971"). InnerVolumeSpecName "kube-api-access-sx2gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:23.168807 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.168768    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9f4532-6b89-474d-be88-8395dfbdb971-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:13:23.168807 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.168800    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2gr\" (UniqueName: \"kubernetes.io/projected/ee9f4532-6b89-474d-be88-8395dfbdb971-kube-api-access-sx2gr\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:13:23.827751 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.827717    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"
Oct 30 23:13:23.827948 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.827745    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d" event={"ID":"ee9f4532-6b89-474d-be88-8395dfbdb971","Type":"ContainerDied","Data":"cd8cac44ea93dd00ea01fe8955332fdc93694f7a7b9888dba42d082e6b9c89d1"}
Oct 30 23:13:23.827948 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:23.827801    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd8cac44ea93dd00ea01fe8955332fdc93694f7a7b9888dba42d082e6b9c89d1"
Oct 30 23:13:37.033069 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:37.032987    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"]
Oct 30 23:13:37.041816 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:37.041790    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s6mzl"]
Oct 30 23:13:38.095414 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.095376    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb" path="/var/lib/kubelet/pods/a6d10d90-8815-4f2e-ba2c-a55e5cb0bfdb/volumes"
Oct 30 23:13:38.246803 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.246762    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"]
Oct 30 23:13:38.247018 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.246994    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq" podUID="1b1f1049-4f70-4093-8cb8-8081e40c1e1e" containerName="manager" containerID="cri-o://a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87" gracePeriod=10
Oct 30 23:13:38.515268 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.515241    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 23:13:38.532761 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.532731    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-extensions-socket-volume\") pod \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\" (UID: \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\") "
Oct 30 23:13:38.532917 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.532773    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-kube-api-access-4gln7\" (UniqueName: \"kubernetes.io/projected/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-kube-api-access-4gln7\") pod \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\" (UID: \"1b1f1049-4f70-4093-8cb8-8081e40c1e1e\") "
Oct 30 23:13:38.533156 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.533127    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-extensions-socket-volume" pod "1b1f1049-4f70-4093-8cb8-8081e40c1e1e" (UID: "1b1f1049-4f70-4093-8cb8-8081e40c1e1e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:13:38.533258 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.533240    4581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-extensions-socket-volume\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:13:38.534993 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.534972    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-kube-api-access-4gln7" pod "1b1f1049-4f70-4093-8cb8-8081e40c1e1e" (UID: "1b1f1049-4f70-4093-8cb8-8081e40c1e1e"). InnerVolumeSpecName "kube-api-access-4gln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:13:38.633870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.633776    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gln7\" (UniqueName: \"kubernetes.io/projected/1b1f1049-4f70-4093-8cb8-8081e40c1e1e-kube-api-access-4gln7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:13:38.923584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.923483    4581 generic.go:356] "Generic (PLEG): container finished" podID="1b1f1049-4f70-4093-8cb8-8081e40c1e1e" containerID="a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87" exitCode=0
Oct 30 23:13:38.923584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.923535    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"
Oct 30 23:13:38.923584 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.923578    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq" event={"ID":"1b1f1049-4f70-4093-8cb8-8081e40c1e1e","Type":"ContainerDied","Data":"a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87"}
Oct 30 23:13:38.923904 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.923625    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq" event={"ID":"1b1f1049-4f70-4093-8cb8-8081e40c1e1e","Type":"ContainerDied","Data":"1e98b7fce969078415cd55f14db87219c255832d999bfc7c4568eec904dadba2"}
Oct 30 23:13:38.923904 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.923646    4581 scope.go:122] "RemoveContainer" containerID="a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87"
Oct 30 23:13:38.936870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.936850    4581 scope.go:122] "RemoveContainer" containerID="a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87"
Oct 30 23:13:38.937169 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:13:38.937148    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87\": container with ID starting with a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87 not found: ID does not exist" containerID="a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87"
Oct 30 23:13:38.937251 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.937180    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87"} err="failed to get container status \"a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87\": rpc error: code = NotFound desc = could not find container \"a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87\": container with ID starting with a254dfa099c8b34ceae20cfab7533f3210e2b68ed2ab27e2cb88d4df727b3e87 not found: ID does not exist"
Oct 30 23:13:38.971525 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.971484    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"]
Oct 30 23:13:38.975488 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:38.975460    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hnqcq"]
Oct 30 23:13:40.096303 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:40.096260    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1f1049-4f70-4093-8cb8-8081e40c1e1e" path="/var/lib/kubelet/pods/1b1f1049-4f70-4093-8cb8-8081e40c1e1e/volumes"
Oct 30 23:13:40.816843 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:13:40.816807    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:13:48.032663 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:48.032624    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"]
Oct 30 23:13:48.041715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:48.041687    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-74tkr"]
Oct 30 23:13:48.096041 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:13:48.096013    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1151835-e326-44c0-a38e-bc2acb33293c" path="/var/lib/kubelet/pods/e1151835-e326-44c0-a38e-bc2acb33293c/volumes"
Oct 30 23:14:05.379504 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:05.379470    4581 scope.go:122] "RemoveContainer" containerID="5878617809f80de65105f027f075949bf8277d2bcffab1d03e83f75d656213e2"
Oct 30 23:14:05.389210 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:05.389186    4581 scope.go:122] "RemoveContainer" containerID="11c00c1b313c8be685114481fd2f7e8ee85d9aec66e2e63eb8bdaaca0216906b"
Oct 30 23:14:08.028263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:08.028209    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"]
Oct 30 23:14:08.037273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:08.037212    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2vnxc"]
Oct 30 23:14:08.094890 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:08.094852    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7badff-2f63-4a26-ba3e-87ad50fd4d18" path="/var/lib/kubelet/pods/6c7badff-2f63-4a26-ba3e-87ad50fd4d18/volumes"
Oct 30 23:14:10.816238 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:14:10.816185    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:14:30.031896 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:30.031860    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"]
Oct 30 23:14:30.041340 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:30.041310    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qqqsj"]
Oct 30 23:14:30.095182 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:30.095150    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbbbe7a-82a7-49bc-9995-3e7bdb083c39" path="/var/lib/kubelet/pods/0cbbbe7a-82a7-49bc-9995-3e7bdb083c39/volumes"
Oct 30 23:14:40.816266 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:14:40.816204    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:14:41.396317 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.396276    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"]
Oct 30 23:14:41.396705 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.396692    4581 cpu_manager.go:414] "containerMap: removing container" podUID="1b1f1049-4f70-4093-8cb8-8081e40c1e1e" containerName="manager"
Oct 30 23:14:41.396758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.396707    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1f1049-4f70-4093-8cb8-8081e40c1e1e" containerName="manager"
Oct 30 23:14:41.396758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.396725    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ee9f4532-6b89-474d-be88-8395dfbdb971" containerName="revoke-keys"
Oct 30 23:14:41.396758 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.396730    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9f4532-6b89-474d-be88-8395dfbdb971" containerName="revoke-keys"
Oct 30 23:14:41.396861 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.396791    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1f1049-4f70-4093-8cb8-8081e40c1e1e" containerName="manager"
Oct 30 23:14:41.396861 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.396802    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9f4532-6b89-474d-be88-8395dfbdb971" containerName="revoke-keys"
Oct 30 23:14:41.399978 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.399953    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:41.415989 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.415968    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bwgbh\""
Oct 30 23:14:41.428672 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.428640    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"]
Oct 30 23:14:41.542073 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.542034    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43439f21-b41f-4d69-bcbc-8677b0082845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l5gbl\" (UID: \"43439f21-b41f-4d69-bcbc-8677b0082845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:41.542288 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.542183    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbw76\" (UniqueName: \"kubernetes.io/projected/43439f21-b41f-4d69-bcbc-8677b0082845-kube-api-access-sbw76\") pod \"kuadrant-operator-controller-manager-678d87766f-l5gbl\" (UID: \"43439f21-b41f-4d69-bcbc-8677b0082845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:41.643388 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.643345    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbw76\" (UniqueName: \"kubernetes.io/projected/43439f21-b41f-4d69-bcbc-8677b0082845-kube-api-access-sbw76\") pod \"kuadrant-operator-controller-manager-678d87766f-l5gbl\" (UID: \"43439f21-b41f-4d69-bcbc-8677b0082845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:41.643572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.643416    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43439f21-b41f-4d69-bcbc-8677b0082845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l5gbl\" (UID: \"43439f21-b41f-4d69-bcbc-8677b0082845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:41.643767 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.643751    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43439f21-b41f-4d69-bcbc-8677b0082845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l5gbl\" (UID: \"43439f21-b41f-4d69-bcbc-8677b0082845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:41.661338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.661268    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbw76\" (UniqueName: \"kubernetes.io/projected/43439f21-b41f-4d69-bcbc-8677b0082845-kube-api-access-sbw76\") pod \"kuadrant-operator-controller-manager-678d87766f-l5gbl\" (UID: \"43439f21-b41f-4d69-bcbc-8677b0082845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:41.712697 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:41.712662    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:42.045026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:42.044938    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"]
Oct 30 23:14:42.048034 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:14:42.048003    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43439f21_b41f_4d69_bcbc_8677b0082845.slice/crio-ddf906370d96e229f6f41023d52ebf77d1c2e46360f28b8bc40c129262569498 WatchSource:0}: Error finding container ddf906370d96e229f6f41023d52ebf77d1c2e46360f28b8bc40c129262569498: Status 404 returned error can't find the container with id ddf906370d96e229f6f41023d52ebf77d1c2e46360f28b8bc40c129262569498
Oct 30 23:14:42.305852 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:42.305755    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl" event={"ID":"43439f21-b41f-4d69-bcbc-8677b0082845","Type":"ContainerStarted","Data":"aea17fedb068094266504d26b8eb167005428d0b8e87cd5d18b047118b8e0945"}
Oct 30 23:14:42.305852 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:42.305823    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:14:42.305852 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:42.305846    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl" event={"ID":"43439f21-b41f-4d69-bcbc-8677b0082845","Type":"ContainerStarted","Data":"ddf906370d96e229f6f41023d52ebf77d1c2e46360f28b8bc40c129262569498"}
Oct 30 23:14:42.347070 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:42.347014    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl" podStartSLOduration=1.346999423 podStartE2EDuration="1.346999423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:42.345103543 +0000 UTC m=+4066.384042426" watchObservedRunningTime="2026-10-30 23:14:42.346999423 +0000 UTC m=+4066.385938308"
Oct 30 23:14:51.028689 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:51.028653    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"]
Oct 30 23:14:51.038322 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:51.038286    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6dptq"]
Oct 30 23:14:52.094646 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:52.094615    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6d2859-c74f-4b86-ab28-ac7412a412ed" path="/var/lib/kubelet/pods/8c6d2859-c74f-4b86-ab28-ac7412a412ed/volumes"
Oct 30 23:14:53.314431 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:14:53.314406    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l5gbl"
Oct 30 23:15:00.148498 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.148450    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"]
Oct 30 23:15:00.152564 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.152537    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:00.164825 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.164800    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:15:00.174571 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.174542    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:15:00.174813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.174794    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:15:00.179310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.179282    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"]
Oct 30 23:15:00.215473 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.215432    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt79t\" (UniqueName: \"kubernetes.io/projected/ce24beef-9629-4934-a189-b153e7776e17-kube-api-access-bt79t\") pod \"maas-api-key-cleanup-29890035-9m5xt\" (UID: \"ce24beef-9629-4934-a189-b153e7776e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:00.215661 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.215481    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce24beef-9629-4934-a189-b153e7776e17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890035-9m5xt\" (UID: \"ce24beef-9629-4934-a189-b153e7776e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:00.316537 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.316499    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt79t\" (UniqueName: \"kubernetes.io/projected/ce24beef-9629-4934-a189-b153e7776e17-kube-api-access-bt79t\") pod \"maas-api-key-cleanup-29890035-9m5xt\" (UID: \"ce24beef-9629-4934-a189-b153e7776e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:00.316766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.316546    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce24beef-9629-4934-a189-b153e7776e17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890035-9m5xt\" (UID: \"ce24beef-9629-4934-a189-b153e7776e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:00.317236 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.317198    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce24beef-9629-4934-a189-b153e7776e17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890035-9m5xt\" (UID: \"ce24beef-9629-4934-a189-b153e7776e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:00.333886 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.333859    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt79t\" (UniqueName: \"kubernetes.io/projected/ce24beef-9629-4934-a189-b153e7776e17-kube-api-access-bt79t\") pod \"maas-api-key-cleanup-29890035-9m5xt\" (UID: \"ce24beef-9629-4934-a189-b153e7776e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:00.466966 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.466884    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:00.775684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:00.775645    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"]
Oct 30 23:15:00.777403 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:15:00.777370    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce24beef_9629_4934_a189_b153e7776e17.slice/crio-9acd857728f1d010a415de08fd67cf6f9a5b606f680ba689ae0005f3ade44681 WatchSource:0}: Error finding container 9acd857728f1d010a415de08fd67cf6f9a5b606f680ba689ae0005f3ade44681: Status 404 returned error can't find the container with id 9acd857728f1d010a415de08fd67cf6f9a5b606f680ba689ae0005f3ade44681
Oct 30 23:15:01.435242 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:01.434665    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" event={"ID":"ce24beef-9629-4934-a189-b153e7776e17","Type":"ContainerStarted","Data":"63500b92cdfaf79a3879e99bdf14d10e584cc08973a16ed2f1ad7f465e8520ea"}
Oct 30 23:15:01.435242 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:01.434741    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" event={"ID":"ce24beef-9629-4934-a189-b153e7776e17","Type":"ContainerStarted","Data":"9acd857728f1d010a415de08fd67cf6f9a5b606f680ba689ae0005f3ade44681"}
Oct 30 23:15:01.471745 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:01.471676    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" podStartSLOduration=1.471654512 podStartE2EDuration="1.471654512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:01.467546231 +0000 UTC m=+4085.506485157" watchObservedRunningTime="2026-10-30 23:15:01.471654512 +0000 UTC m=+4085.510593399"
Oct 30 23:15:02.480763 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:02.479523    4581 generic.go:356] "Generic (PLEG): container finished" podID="ce24beef-9629-4934-a189-b153e7776e17" containerID="63500b92cdfaf79a3879e99bdf14d10e584cc08973a16ed2f1ad7f465e8520ea" exitCode=60
Oct 30 23:15:02.480763 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:02.479675    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" event={"ID":"ce24beef-9629-4934-a189-b153e7776e17","Type":"ContainerDied","Data":"63500b92cdfaf79a3879e99bdf14d10e584cc08973a16ed2f1ad7f465e8520ea"}
Oct 30 23:15:02.480763 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:02.480083    4581 scope.go:122] "RemoveContainer" containerID="63500b92cdfaf79a3879e99bdf14d10e584cc08973a16ed2f1ad7f465e8520ea"
Oct 30 23:15:03.506250 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:03.505629    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" event={"ID":"ce24beef-9629-4934-a189-b153e7776e17","Type":"ContainerStarted","Data":"2481d01a65126042e420e3fa09e55d701132d67471b7d5c4ce25bf6e8dc0f767"}
Oct 30 23:15:04.513741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:04.513705    4581 generic.go:356] "Generic (PLEG): container finished" podID="ce24beef-9629-4934-a189-b153e7776e17" containerID="2481d01a65126042e420e3fa09e55d701132d67471b7d5c4ce25bf6e8dc0f767" exitCode=60
Oct 30 23:15:04.514274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:04.513787    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" event={"ID":"ce24beef-9629-4934-a189-b153e7776e17","Type":"ContainerDied","Data":"2481d01a65126042e420e3fa09e55d701132d67471b7d5c4ce25bf6e8dc0f767"}
Oct 30 23:15:04.514274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:04.513858    4581 scope.go:122] "RemoveContainer" containerID="63500b92cdfaf79a3879e99bdf14d10e584cc08973a16ed2f1ad7f465e8520ea"
Oct 30 23:15:04.514274 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:04.513998    4581 scope.go:122] "RemoveContainer" containerID="2481d01a65126042e420e3fa09e55d701132d67471b7d5c4ce25bf6e8dc0f767"
Oct 30 23:15:04.514274 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:15:04.514262    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890035-9m5xt_odh-ai-gateway-infra(ce24beef-9629-4934-a189-b153e7776e17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" podUID="ce24beef-9629-4934-a189-b153e7776e17"
Oct 30 23:15:05.423980 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:05.423949    4581 scope.go:122] "RemoveContainer" containerID="03610b0e060987035a2048cd86364e7707be96be13bcf50cfbc69e24ee6ec8fd"
Oct 30 23:15:05.433555 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:05.433535    4581 scope.go:122] "RemoveContainer" containerID="e28d49feb73782b17946930ebeaa40fbe2b5d2168bd63a3dd7b2f5b8a436c57c"
Oct 30 23:15:05.442623 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:05.442601    4581 scope.go:122] "RemoveContainer" containerID="312fb099d11d590e25c8140276e0fcfbcfd8dc01abdaab800571dab5c4914d91"
Oct 30 23:15:05.521954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:05.521924    4581 scope.go:122] "RemoveContainer" containerID="2481d01a65126042e420e3fa09e55d701132d67471b7d5c4ce25bf6e8dc0f767"
Oct 30 23:15:05.522383 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:15:05.522150    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890035-9m5xt_odh-ai-gateway-infra(ce24beef-9629-4934-a189-b153e7776e17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" podUID="ce24beef-9629-4934-a189-b153e7776e17"
Oct 30 23:15:10.815924 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:15:10.815834    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:15:11.031154 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:11.031112    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"]
Oct 30 23:15:11.040363 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:11.040329    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnbtm"]
Oct 30 23:15:12.094717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:12.094681    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2d3b58-ee5b-404d-a4f8-894cbcde91f7" path="/var/lib/kubelet/pods/1f2d3b58-ee5b-404d-a4f8-894cbcde91f7/volumes"
Oct 30 23:15:16.092263 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:16.092228    4581 scope.go:122] "RemoveContainer" containerID="2481d01a65126042e420e3fa09e55d701132d67471b7d5c4ce25bf6e8dc0f767"
Oct 30 23:15:16.596845 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:16.596798    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" event={"ID":"ce24beef-9629-4934-a189-b153e7776e17","Type":"ContainerStarted","Data":"8314bd1d39f1fdd6283e472c07e8c3c9978f97eb05012109dc2e87359a7a7cff"}
Oct 30 23:15:17.605477 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:17.605438    4581 generic.go:356] "Generic (PLEG): container finished" podID="ce24beef-9629-4934-a189-b153e7776e17" containerID="8314bd1d39f1fdd6283e472c07e8c3c9978f97eb05012109dc2e87359a7a7cff" exitCode=60
Oct 30 23:15:17.605919 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:17.605506    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" event={"ID":"ce24beef-9629-4934-a189-b153e7776e17","Type":"ContainerDied","Data":"8314bd1d39f1fdd6283e472c07e8c3c9978f97eb05012109dc2e87359a7a7cff"}
Oct 30 23:15:17.605919 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:17.605561    4581 scope.go:122] "RemoveContainer" containerID="2481d01a65126042e420e3fa09e55d701132d67471b7d5c4ce25bf6e8dc0f767"
Oct 30 23:15:17.606026 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:17.605960    4581 scope.go:122] "RemoveContainer" containerID="8314bd1d39f1fdd6283e472c07e8c3c9978f97eb05012109dc2e87359a7a7cff"
Oct 30 23:15:17.606268 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:15:17.606249    4581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890035-9m5xt_odh-ai-gateway-infra(ce24beef-9629-4934-a189-b153e7776e17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" podUID="ce24beef-9629-4934-a189-b153e7776e17"
Oct 30 23:15:17.645668 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:17.645637    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"]
Oct 30 23:15:18.753595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:18.753567    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:18.852082 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:18.852040    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce24beef-9629-4934-a189-b153e7776e17-kube-api-access-bt79t\" (UniqueName: \"kubernetes.io/projected/ce24beef-9629-4934-a189-b153e7776e17-kube-api-access-bt79t\") pod \"ce24beef-9629-4934-a189-b153e7776e17\" (UID: \"ce24beef-9629-4934-a189-b153e7776e17\") "
Oct 30 23:15:18.852316 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:18.852126    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce24beef-9629-4934-a189-b153e7776e17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce24beef-9629-4934-a189-b153e7776e17-maas-api-service-ca\") pod \"ce24beef-9629-4934-a189-b153e7776e17\" (UID: \"ce24beef-9629-4934-a189-b153e7776e17\") "
Oct 30 23:15:18.852519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:18.852492    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce24beef-9629-4934-a189-b153e7776e17-maas-api-service-ca" pod "ce24beef-9629-4934-a189-b153e7776e17" (UID: "ce24beef-9629-4934-a189-b153e7776e17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:15:18.852703 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:18.852683    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce24beef-9629-4934-a189-b153e7776e17-maas-api-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:15:18.854533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:18.854502    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce24beef-9629-4934-a189-b153e7776e17-kube-api-access-bt79t" pod "ce24beef-9629-4934-a189-b153e7776e17" (UID: "ce24beef-9629-4934-a189-b153e7776e17"). InnerVolumeSpecName "kube-api-access-bt79t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:15:18.953428 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:18.953343    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt79t\" (UniqueName: \"kubernetes.io/projected/ce24beef-9629-4934-a189-b153e7776e17-kube-api-access-bt79t\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:15:19.624984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:19.624945    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"
Oct 30 23:15:19.624984 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:19.624971    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt" event={"ID":"ce24beef-9629-4934-a189-b153e7776e17","Type":"ContainerDied","Data":"9acd857728f1d010a415de08fd67cf6f9a5b606f680ba689ae0005f3ade44681"}
Oct 30 23:15:19.625354 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:19.625053    4581 scope.go:122] "RemoveContainer" containerID="8314bd1d39f1fdd6283e472c07e8c3c9978f97eb05012109dc2e87359a7a7cff"
Oct 30 23:15:19.674124 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:19.674087    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"]
Oct 30 23:15:19.682509 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:19.682479    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890035-9m5xt"]
Oct 30 23:15:20.117244 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:20.106921    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce24beef-9629-4934-a189-b153e7776e17" path="/var/lib/kubelet/pods/ce24beef-9629-4934-a189-b153e7776e17/volumes"
Oct 30 23:15:26.278765 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.278729    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"]
Oct 30 23:15:26.279260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.279109    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:15:26.279260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.279121    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:15:26.279260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.279152    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:15:26.279260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.279157    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:15:26.279260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.279240    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:15:26.279260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.279250    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:15:26.279260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.279259    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:15:26.283458 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.283439    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.296135 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.296109    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:15:26.305922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.305895    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:15:26.306102 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.305896    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:15:26.306102 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.305905    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:15:26.308627 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.308605    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"]
Oct 30 23:15:26.376161 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.376128    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.376369 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.376178    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.376369 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.376210    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8zh\" (UniqueName: \"kubernetes.io/projected/6abb5dd5-080a-4fd1-aec7-67284dc604ac-kube-api-access-7r8zh\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.477292 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.477205    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.477467 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.477319    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.477467 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.477339    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8zh\" (UniqueName: \"kubernetes.io/projected/6abb5dd5-080a-4fd1-aec7-67284dc604ac-kube-api-access-7r8zh\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.478015 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.477981    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.479827 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.479805    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.495715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.495688    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8zh\" (UniqueName: \"kubernetes.io/projected/6abb5dd5-080a-4fd1-aec7-67284dc604ac-kube-api-access-7r8zh\") pod \"maas-api-revoke-sub-d2bc93bc-rfsw9\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.596932 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.596848    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:26.900185 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:26.900150    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"]
Oct 30 23:15:26.903768 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:15:26.903732    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abb5dd5_080a_4fd1_aec7_67284dc604ac.slice/crio-3fac8283a4d062980c9df533efbcabf1616bbe8c69ea0b5a890dd6cdd5780a6a WatchSource:0}: Error finding container 3fac8283a4d062980c9df533efbcabf1616bbe8c69ea0b5a890dd6cdd5780a6a: Status 404 returned error can't find the container with id 3fac8283a4d062980c9df533efbcabf1616bbe8c69ea0b5a890dd6cdd5780a6a
Oct 30 23:15:27.716454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:27.716416    4581 generic.go:356] "Generic (PLEG): container finished" podID="6abb5dd5-080a-4fd1-aec7-67284dc604ac" containerID="51f8545367791fae63223087a5eebedd6df211ec4e1d4074e92190c7044cc1b0" exitCode=0
Oct 30 23:15:27.716917 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:27.716502    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9" event={"ID":"6abb5dd5-080a-4fd1-aec7-67284dc604ac","Type":"ContainerDied","Data":"51f8545367791fae63223087a5eebedd6df211ec4e1d4074e92190c7044cc1b0"}
Oct 30 23:15:27.716917 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:27.716561    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9" event={"ID":"6abb5dd5-080a-4fd1-aec7-67284dc604ac","Type":"ContainerStarted","Data":"3fac8283a4d062980c9df533efbcabf1616bbe8c69ea0b5a890dd6cdd5780a6a"}
Oct 30 23:15:28.867743 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:28.867713    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:29.001940 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.001840    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-xks-ca\") pod \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") "
Oct 30 23:15:29.001940 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.001914    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-openshift-ca\") pod \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") "
Oct 30 23:15:29.002201 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.001955    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abb5dd5-080a-4fd1-aec7-67284dc604ac-kube-api-access-7r8zh\" (UniqueName: \"kubernetes.io/projected/6abb5dd5-080a-4fd1-aec7-67284dc604ac-kube-api-access-7r8zh\") pod \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\" (UID: \"6abb5dd5-080a-4fd1-aec7-67284dc604ac\") "
Oct 30 23:15:29.002325 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.002301    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-openshift-ca" pod "6abb5dd5-080a-4fd1-aec7-67284dc604ac" (UID: "6abb5dd5-080a-4fd1-aec7-67284dc604ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:15:29.002440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.002418    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:15:29.004178 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.004156    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-xks-ca" pod "6abb5dd5-080a-4fd1-aec7-67284dc604ac" (UID: "6abb5dd5-080a-4fd1-aec7-67284dc604ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:15:29.004260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.004233    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abb5dd5-080a-4fd1-aec7-67284dc604ac-kube-api-access-7r8zh" pod "6abb5dd5-080a-4fd1-aec7-67284dc604ac" (UID: "6abb5dd5-080a-4fd1-aec7-67284dc604ac"). InnerVolumeSpecName "kube-api-access-7r8zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:15:29.103971 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.103937    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r8zh\" (UniqueName: \"kubernetes.io/projected/6abb5dd5-080a-4fd1-aec7-67284dc604ac-kube-api-access-7r8zh\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:15:29.103971 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.103967    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6abb5dd5-080a-4fd1-aec7-67284dc604ac-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:15:29.730052 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.730020    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"
Oct 30 23:15:29.730052 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.730040    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9" event={"ID":"6abb5dd5-080a-4fd1-aec7-67284dc604ac","Type":"ContainerDied","Data":"3fac8283a4d062980c9df533efbcabf1616bbe8c69ea0b5a890dd6cdd5780a6a"}
Oct 30 23:15:29.730366 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:15:29.730086    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fac8283a4d062980c9df533efbcabf1616bbe8c69ea0b5a890dd6cdd5780a6a"
Oct 30 23:15:40.817944 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:15:40.817907    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:16:05.477796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:05.477765    4581 scope.go:122] "RemoveContainer" containerID="c2a4731f550382bac4482efdc1038d289c93b6b293a9260aa038f838ab6fd45e"
Oct 30 23:16:05.488669 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:05.488643    4581 scope.go:122] "RemoveContainer" containerID="8011aeefe20d7ef3bd5ea283d541b0bb101d6a1a07f77facf0caae3c7eb0e35c"
Oct 30 23:16:05.498602 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:05.498286    4581 scope.go:122] "RemoveContainer" containerID="e5f162049bbbd55b28052ce1ba6569c49b1918a92acfb33d3492da6a04ad3436"
Oct 30 23:16:10.816673 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:16:10.816631    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:16:12.031909 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:12.031874    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"]
Oct 30 23:16:12.040983 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:12.040949    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gr6mf"]
Oct 30 23:16:12.094741 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:12.094708    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23a0489-72af-4919-808c-17cb7f818cb0" path="/var/lib/kubelet/pods/c23a0489-72af-4919-808c-17cb7f818cb0/volumes"
Oct 30 23:16:24.029840 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:24.029804    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"]
Oct 30 23:16:24.038639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:24.038601    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6t9mr"]
Oct 30 23:16:24.095013 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:24.094978    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ff7db6-fea4-4170-9fae-b6d963faeff3" path="/var/lib/kubelet/pods/46ff7db6-fea4-4170-9fae-b6d963faeff3/volumes"
Oct 30 23:16:31.739470 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.739431    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"]
Oct 30 23:16:31.739874 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.739831    4581 cpu_manager.go:414] "containerMap: removing container" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:16:31.739874 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.739842    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce24beef-9629-4934-a189-b153e7776e17" containerName="cleanup"
Oct 30 23:16:31.739874 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.739870    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6abb5dd5-080a-4fd1-aec7-67284dc604ac" containerName="revoke-keys"
Oct 30 23:16:31.739874 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.739875    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abb5dd5-080a-4fd1-aec7-67284dc604ac" containerName="revoke-keys"
Oct 30 23:16:31.740019 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.739945    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abb5dd5-080a-4fd1-aec7-67284dc604ac" containerName="revoke-keys"
Oct 30 23:16:31.744772 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.744753    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.756974 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.756955    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:16:31.766658 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.766632    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:16:31.766814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.766681    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:16:31.766814 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.766710    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:16:31.769685 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.769657    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"]
Oct 30 23:16:31.836954 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.836911    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qn6\" (UniqueName: \"kubernetes.io/projected/56acd89d-7426-4525-8813-2384c2220559-kube-api-access-q4qn6\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.837169 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.837078    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56acd89d-7426-4525-8813-2384c2220559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.837169 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.837153    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56acd89d-7426-4525-8813-2384c2220559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.937973 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.937927    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56acd89d-7426-4525-8813-2384c2220559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.938171 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.937981    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56acd89d-7426-4525-8813-2384c2220559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.938171 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.938062    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qn6\" (UniqueName: \"kubernetes.io/projected/56acd89d-7426-4525-8813-2384c2220559-kube-api-access-q4qn6\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.938698 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.938671    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56acd89d-7426-4525-8813-2384c2220559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.940608 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.940586    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56acd89d-7426-4525-8813-2384c2220559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:31.956842 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:31.956808    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qn6\" (UniqueName: \"kubernetes.io/projected/56acd89d-7426-4525-8813-2384c2220559-kube-api-access-q4qn6\") pod \"maas-api-revoke-sub-a04f1b3c-9nn7m\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:32.058562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:32.058481    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:32.370254 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:32.370201    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"]
Oct 30 23:16:32.373726 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:16:32.373699    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56acd89d_7426_4525_8813_2384c2220559.slice/crio-278bf3ea14c1042f6d00374147f47fdbd4188a62be091b2bfa6c1b68001f28a0 WatchSource:0}: Error finding container 278bf3ea14c1042f6d00374147f47fdbd4188a62be091b2bfa6c1b68001f28a0: Status 404 returned error can't find the container with id 278bf3ea14c1042f6d00374147f47fdbd4188a62be091b2bfa6c1b68001f28a0
Oct 30 23:16:33.134822 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:33.134790    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m" event={"ID":"56acd89d-7426-4525-8813-2384c2220559","Type":"ContainerStarted","Data":"83300ad8c1add5a9bcd1c855069bda3712ace5300a28785b197acb5e5bd642d7"}
Oct 30 23:16:33.135295 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:33.134848    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m" event={"ID":"56acd89d-7426-4525-8813-2384c2220559","Type":"ContainerStarted","Data":"278bf3ea14c1042f6d00374147f47fdbd4188a62be091b2bfa6c1b68001f28a0"}
Oct 30 23:16:33.171115 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:33.171043    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m" podStartSLOduration=2.171024199 podStartE2EDuration="2.171024199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:16:33.169446602 +0000 UTC m=+4177.208385487" watchObservedRunningTime="2026-10-30 23:16:33.171024199 +0000 UTC m=+4177.209963086"
Oct 30 23:16:34.143848 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:34.143811    4581 generic.go:356] "Generic (PLEG): container finished" podID="56acd89d-7426-4525-8813-2384c2220559" containerID="83300ad8c1add5a9bcd1c855069bda3712ace5300a28785b197acb5e5bd642d7" exitCode=0
Oct 30 23:16:34.144352 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:34.143905    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m" event={"ID":"56acd89d-7426-4525-8813-2384c2220559","Type":"ContainerDied","Data":"83300ad8c1add5a9bcd1c855069bda3712ace5300a28785b197acb5e5bd642d7"}
Oct 30 23:16:35.289031 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.289008    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:35.372605 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.372566    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56acd89d-7426-4525-8813-2384c2220559-kube-api-access-q4qn6\" (UniqueName: \"kubernetes.io/projected/56acd89d-7426-4525-8813-2384c2220559-kube-api-access-q4qn6\") pod \"56acd89d-7426-4525-8813-2384c2220559\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") "
Oct 30 23:16:35.372813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.372685    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56acd89d-7426-4525-8813-2384c2220559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56acd89d-7426-4525-8813-2384c2220559-maas-api-openshift-ca\") pod \"56acd89d-7426-4525-8813-2384c2220559\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") "
Oct 30 23:16:35.372813 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.372725    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56acd89d-7426-4525-8813-2384c2220559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56acd89d-7426-4525-8813-2384c2220559-maas-api-xks-ca\") pod \"56acd89d-7426-4525-8813-2384c2220559\" (UID: \"56acd89d-7426-4525-8813-2384c2220559\") "
Oct 30 23:16:35.373124 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.373094    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56acd89d-7426-4525-8813-2384c2220559-maas-api-openshift-ca" pod "56acd89d-7426-4525-8813-2384c2220559" (UID: "56acd89d-7426-4525-8813-2384c2220559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:16:35.374884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.374852    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56acd89d-7426-4525-8813-2384c2220559-kube-api-access-q4qn6" pod "56acd89d-7426-4525-8813-2384c2220559" (UID: "56acd89d-7426-4525-8813-2384c2220559"). InnerVolumeSpecName "kube-api-access-q4qn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:16:35.375033 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.374985    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56acd89d-7426-4525-8813-2384c2220559-maas-api-xks-ca" pod "56acd89d-7426-4525-8813-2384c2220559" (UID: "56acd89d-7426-4525-8813-2384c2220559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:16:35.474174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.474078    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qn6\" (UniqueName: \"kubernetes.io/projected/56acd89d-7426-4525-8813-2384c2220559-kube-api-access-q4qn6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:16:35.474174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.474112    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56acd89d-7426-4525-8813-2384c2220559-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:16:35.474174 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:35.474122    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56acd89d-7426-4525-8813-2384c2220559-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:16:36.159735 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:36.159652    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m" event={"ID":"56acd89d-7426-4525-8813-2384c2220559","Type":"ContainerDied","Data":"278bf3ea14c1042f6d00374147f47fdbd4188a62be091b2bfa6c1b68001f28a0"}
Oct 30 23:16:36.159735 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:36.159684    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"
Oct 30 23:16:36.159735 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:36.159708    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278bf3ea14c1042f6d00374147f47fdbd4188a62be091b2bfa6c1b68001f28a0"
Oct 30 23:16:40.816580 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:16:40.816535    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:16:42.971623 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.971580    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"]
Oct 30 23:16:42.972208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.972186    4581 cpu_manager.go:414] "containerMap: removing container" podUID="56acd89d-7426-4525-8813-2384c2220559" containerName="revoke-keys"
Oct 30 23:16:42.972208 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.972211    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56acd89d-7426-4525-8813-2384c2220559" containerName="revoke-keys"
Oct 30 23:16:42.972413 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.972327    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="56acd89d-7426-4525-8813-2384c2220559" containerName="revoke-keys"
Oct 30 23:16:42.975647 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.975624    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:42.987854 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.987830    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:16:42.997607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.997578    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:16:42.997607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.997582    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:16:42.997836 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:42.997620    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:16:43.000872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.000847    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"]
Oct 30 23:16:43.055103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.055061    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.055310 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.055208    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.055364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.055308    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkx6\" (UniqueName: \"kubernetes.io/projected/52178043-14cb-45be-ac69-c36ca7905aa8-kube-api-access-pfkx6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.156281 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.156247    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkx6\" (UniqueName: \"kubernetes.io/projected/52178043-14cb-45be-ac69-c36ca7905aa8-kube-api-access-pfkx6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.156544 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.156319    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.156544 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.156386    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.157159 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.157130    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.159268 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.159239    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.175823 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.175797    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkx6\" (UniqueName: \"kubernetes.io/projected/52178043-14cb-45be-ac69-c36ca7905aa8-kube-api-access-pfkx6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.292457 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.292363    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:43.601287 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:43.600751    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"]
Oct 30 23:16:43.606715 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:16:43.606678    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52178043_14cb_45be_ac69_c36ca7905aa8.slice/crio-c05ed3c30fd5ff36157990ebe654c5838a2416eeb8e27809a9d4506f136d6cf8 WatchSource:0}: Error finding container c05ed3c30fd5ff36157990ebe654c5838a2416eeb8e27809a9d4506f136d6cf8: Status 404 returned error can't find the container with id c05ed3c30fd5ff36157990ebe654c5838a2416eeb8e27809a9d4506f136d6cf8
Oct 30 23:16:44.209718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:44.209622    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5" event={"ID":"52178043-14cb-45be-ac69-c36ca7905aa8","Type":"ContainerStarted","Data":"9502eeb28774eb72301c4c047ab644ec8a3d3c285f25af71d0bf3db546c9ce8a"}
Oct 30 23:16:44.209718 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:44.209681    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5" event={"ID":"52178043-14cb-45be-ac69-c36ca7905aa8","Type":"ContainerStarted","Data":"c05ed3c30fd5ff36157990ebe654c5838a2416eeb8e27809a9d4506f136d6cf8"}
Oct 30 23:16:44.245685 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:44.245623    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5" podStartSLOduration=2.245609436 podStartE2EDuration="2.245609436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:16:44.24504941 +0000 UTC m=+4188.283988308" watchObservedRunningTime="2026-10-30 23:16:44.245609436 +0000 UTC m=+4188.284548321"
Oct 30 23:16:45.219123 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:45.219082    4581 generic.go:356] "Generic (PLEG): container finished" podID="52178043-14cb-45be-ac69-c36ca7905aa8" containerID="9502eeb28774eb72301c4c047ab644ec8a3d3c285f25af71d0bf3db546c9ce8a" exitCode=0
Oct 30 23:16:45.219593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:45.219183    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5" event={"ID":"52178043-14cb-45be-ac69-c36ca7905aa8","Type":"ContainerDied","Data":"9502eeb28774eb72301c4c047ab644ec8a3d3c285f25af71d0bf3db546c9ce8a"}
Oct 30 23:16:46.362537 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.362505    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:46.491093 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.491006    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-openshift-ca\") pod \"52178043-14cb-45be-ac69-c36ca7905aa8\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") "
Oct 30 23:16:46.491093 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.491047    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-xks-ca\") pod \"52178043-14cb-45be-ac69-c36ca7905aa8\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") "
Oct 30 23:16:46.491093 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.491096    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52178043-14cb-45be-ac69-c36ca7905aa8-kube-api-access-pfkx6\" (UniqueName: \"kubernetes.io/projected/52178043-14cb-45be-ac69-c36ca7905aa8-kube-api-access-pfkx6\") pod \"52178043-14cb-45be-ac69-c36ca7905aa8\" (UID: \"52178043-14cb-45be-ac69-c36ca7905aa8\") "
Oct 30 23:16:46.491507 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.491479    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-openshift-ca" pod "52178043-14cb-45be-ac69-c36ca7905aa8" (UID: "52178043-14cb-45be-ac69-c36ca7905aa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:16:46.493454 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.493415    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-xks-ca" pod "52178043-14cb-45be-ac69-c36ca7905aa8" (UID: "52178043-14cb-45be-ac69-c36ca7905aa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:16:46.493586 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.493550    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52178043-14cb-45be-ac69-c36ca7905aa8-kube-api-access-pfkx6" pod "52178043-14cb-45be-ac69-c36ca7905aa8" (UID: "52178043-14cb-45be-ac69-c36ca7905aa8"). InnerVolumeSpecName "kube-api-access-pfkx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:16:46.592551 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.592513    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:16:46.592551 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.592545    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52178043-14cb-45be-ac69-c36ca7905aa8-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:16:46.592551 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:46.592554    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfkx6\" (UniqueName: \"kubernetes.io/projected/52178043-14cb-45be-ac69-c36ca7905aa8-kube-api-access-pfkx6\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:16:47.233778 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:47.233733    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5" event={"ID":"52178043-14cb-45be-ac69-c36ca7905aa8","Type":"ContainerDied","Data":"c05ed3c30fd5ff36157990ebe654c5838a2416eeb8e27809a9d4506f136d6cf8"}
Oct 30 23:16:47.234009 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:47.233800    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05ed3c30fd5ff36157990ebe654c5838a2416eeb8e27809a9d4506f136d6cf8"
Oct 30 23:16:47.234009 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:47.233755    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"
Oct 30 23:16:55.580370 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:55.580334    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"]
Oct 30 23:16:55.600111 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:55.600078    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ef9544-jzbk5"]
Oct 30 23:16:56.096498 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:56.096463    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52178043-14cb-45be-ac69-c36ca7905aa8" path="/var/lib/kubelet/pods/52178043-14cb-45be-ac69-c36ca7905aa8/volumes"
Oct 30 23:16:57.646071 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:57.646038    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:16:57.655432 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:16:57.655404    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:17:05.535533 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:05.535499    4581 scope.go:122] "RemoveContainer" containerID="c04cc7577efe4b4280bfda9514e22de2c6270bd1f3a69db73a342424b9e83862"
Oct 30 23:17:05.544850 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:05.544827    4581 scope.go:122] "RemoveContainer" containerID="17d0efe7bafab190d60a7356d8ba4577e222ffa5d7b6863af93dbd199df195fa"
Oct 30 23:17:05.554010 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:05.553986    4581 scope.go:122] "RemoveContainer" containerID="28d2d83f416fab766e8449241f8619205ea22de85a201d31410dea221cfe8a0b"
Oct 30 23:17:10.815971 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:17:10.815932    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:17:40.817092 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:17:40.817050    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:17:45.828273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.828237    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"]
Oct 30 23:17:45.828815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.828622    4581 cpu_manager.go:414] "containerMap: removing container" podUID="52178043-14cb-45be-ac69-c36ca7905aa8" containerName="revoke-keys"
Oct 30 23:17:45.828815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.828633    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52178043-14cb-45be-ac69-c36ca7905aa8" containerName="revoke-keys"
Oct 30 23:17:45.828815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.828724    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="52178043-14cb-45be-ac69-c36ca7905aa8" containerName="revoke-keys"
Oct 30 23:17:45.832068 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.832042    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:45.853567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.853536    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:17:45.853567 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.853560    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:17:45.853794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.853588    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:17:45.853794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.853542    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:17:45.856366 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.856343    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"]
Oct 30 23:17:45.890685 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.890650    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29fn\" (UniqueName: \"kubernetes.io/projected/66ede7f6-f043-4dee-9191-7492fe5bf7e3-kube-api-access-h29fn\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:45.890859 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.890773    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:45.890907 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.890871    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:45.991967 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.991927    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:45.992166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.991986    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29fn\" (UniqueName: \"kubernetes.io/projected/66ede7f6-f043-4dee-9191-7492fe5bf7e3-kube-api-access-h29fn\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:45.992166 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.992109    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:45.992767 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.992749    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:45.994519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:45.994495    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:46.009354 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:46.009318    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29fn\" (UniqueName: \"kubernetes.io/projected/66ede7f6-f043-4dee-9191-7492fe5bf7e3-kube-api-access-h29fn\") pod \"maas-api-revoke-sub-452b6d7b-sqlpq\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:46.146564 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:46.146521    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:46.446923 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:46.446774    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"]
Oct 30 23:17:46.450212 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:17:46.450181    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ede7f6_f043_4dee_9191_7492fe5bf7e3.slice/crio-fea1bd564b5644e28ce68e60e0bd3700654c06965989013c642558be6e536ef5 WatchSource:0}: Error finding container fea1bd564b5644e28ce68e60e0bd3700654c06965989013c642558be6e536ef5: Status 404 returned error can't find the container with id fea1bd564b5644e28ce68e60e0bd3700654c06965989013c642558be6e536ef5
Oct 30 23:17:46.694524 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:46.694484    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq" event={"ID":"66ede7f6-f043-4dee-9191-7492fe5bf7e3","Type":"ContainerStarted","Data":"fea1bd564b5644e28ce68e60e0bd3700654c06965989013c642558be6e536ef5"}
Oct 30 23:17:47.706863 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:47.706818    4581 generic.go:356] "Generic (PLEG): container finished" podID="66ede7f6-f043-4dee-9191-7492fe5bf7e3" containerID="9dbca3fe12ccd651bfa99c18c72de421a0dcc22f3a71318628368a32db447f8f" exitCode=0
Oct 30 23:17:47.707465 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:47.706919    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq" event={"ID":"66ede7f6-f043-4dee-9191-7492fe5bf7e3","Type":"ContainerDied","Data":"9dbca3fe12ccd651bfa99c18c72de421a0dcc22f3a71318628368a32db447f8f"}
Oct 30 23:17:49.445736 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.445393    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:49.543695 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.543601    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-xks-ca\") pod \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") "
Oct 30 23:17:49.544037 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.544010    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ede7f6-f043-4dee-9191-7492fe5bf7e3-kube-api-access-h29fn\" (UniqueName: \"kubernetes.io/projected/66ede7f6-f043-4dee-9191-7492fe5bf7e3-kube-api-access-h29fn\") pod \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") "
Oct 30 23:17:49.544206 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.544188    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-openshift-ca\") pod \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\" (UID: \"66ede7f6-f043-4dee-9191-7492fe5bf7e3\") "
Oct 30 23:17:49.557548 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.557511    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-openshift-ca" pod "66ede7f6-f043-4dee-9191-7492fe5bf7e3" (UID: "66ede7f6-f043-4dee-9191-7492fe5bf7e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:17:49.575908 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.575854    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ede7f6-f043-4dee-9191-7492fe5bf7e3-kube-api-access-h29fn" pod "66ede7f6-f043-4dee-9191-7492fe5bf7e3" (UID: "66ede7f6-f043-4dee-9191-7492fe5bf7e3"). InnerVolumeSpecName "kube-api-access-h29fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:17:49.588737 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.588676    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-xks-ca" pod "66ede7f6-f043-4dee-9191-7492fe5bf7e3" (UID: "66ede7f6-f043-4dee-9191-7492fe5bf7e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:17:49.650998 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.650891    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:17:49.650998 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.650935    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h29fn\" (UniqueName: \"kubernetes.io/projected/66ede7f6-f043-4dee-9191-7492fe5bf7e3-kube-api-access-h29fn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:17:49.650998 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.650956    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ede7f6-f043-4dee-9191-7492fe5bf7e3-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:17:49.812302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.808838    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq" event={"ID":"66ede7f6-f043-4dee-9191-7492fe5bf7e3","Type":"ContainerDied","Data":"fea1bd564b5644e28ce68e60e0bd3700654c06965989013c642558be6e536ef5"}
Oct 30 23:17:49.812302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.808902    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fea1bd564b5644e28ce68e60e0bd3700654c06965989013c642558be6e536ef5"
Oct 30 23:17:49.812302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:49.808975    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"
Oct 30 23:17:51.036346 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:51.036305    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"]
Oct 30 23:17:51.044312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:51.044245    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a579ea-7s2br"]
Oct 30 23:17:52.096358 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:17:52.096326    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174f3dd5-eb83-4931-b1d4-3f0f140cccaf" path="/var/lib/kubelet/pods/174f3dd5-eb83-4931-b1d4-3f0f140cccaf/volumes"
Oct 30 23:18:02.846640 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:02.846595    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:18:02.847350 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:02.847315    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:18:05.098423 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.098381    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"]
Oct 30 23:18:05.099649 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.099620    4581 cpu_manager.go:414] "containerMap: removing container" podUID="66ede7f6-f043-4dee-9191-7492fe5bf7e3" containerName="revoke-keys"
Oct 30 23:18:05.099818 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.099802    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ede7f6-f043-4dee-9191-7492fe5bf7e3" containerName="revoke-keys"
Oct 30 23:18:05.100065 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.100051    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ede7f6-f043-4dee-9191-7492fe5bf7e3" containerName="revoke-keys"
Oct 30 23:18:05.105699 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.105672    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.123752 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.123716    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"]
Oct 30 23:18:05.128406 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.128374    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:18:05.128573 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.128476    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:18:05.129162 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.129139    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:18:05.129466 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.129446    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:18:05.187425 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.187370    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271aabed-e533-4934-956c-747525e2900b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.187635 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.187485    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271aabed-e533-4934-956c-747525e2900b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.187635 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.187517    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tjc\" (UniqueName: \"kubernetes.io/projected/271aabed-e533-4934-956c-747525e2900b-kube-api-access-t7tjc\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.289336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.289288    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271aabed-e533-4934-956c-747525e2900b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.289650 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.289354    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tjc\" (UniqueName: \"kubernetes.io/projected/271aabed-e533-4934-956c-747525e2900b-kube-api-access-t7tjc\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.289650 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.289477    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271aabed-e533-4934-956c-747525e2900b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.290377 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.290337    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271aabed-e533-4934-956c-747525e2900b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.292787 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.292764    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271aabed-e533-4934-956c-747525e2900b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.311045 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.311013    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tjc\" (UniqueName: \"kubernetes.io/projected/271aabed-e533-4934-956c-747525e2900b-kube-api-access-t7tjc\") pod \"maas-api-revoke-sub-3e31a506-45gj7\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.426335 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.426289    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:05.588460 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.588425    4581 scope.go:122] "RemoveContainer" containerID="de6186773be9edaaa722cb9599298b69d3c7eea7f4e7ef00395da233ab42135e"
Oct 30 23:18:05.649429 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:05.647415    4581 scope.go:122] "RemoveContainer" containerID="7b6c545d7db400ba915686de9aaae5cc6663784cafc047a88b6a5f77b45949fb"
Oct 30 23:18:06.207453 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:18:06.206593    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271aabed_e533_4934_956c_747525e2900b.slice/crio-1bdcfb50e43e7fc491da38aadb15517fc238a2e927901ee120e057e20f0b3086 WatchSource:0}: Error finding container 1bdcfb50e43e7fc491da38aadb15517fc238a2e927901ee120e057e20f0b3086: Status 404 returned error can't find the container with id 1bdcfb50e43e7fc491da38aadb15517fc238a2e927901ee120e057e20f0b3086
Oct 30 23:18:06.208249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:06.207999    4581 scope.go:122] "RemoveContainer" containerID="13c5c6dfa716e6ad64bf35da9ab9ff42587a0dbb8d5722c5e293d8dc8347739d"
Oct 30 23:18:06.238455 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:06.238426    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"]
Oct 30 23:18:07.320042 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:07.320002    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7" event={"ID":"271aabed-e533-4934-956c-747525e2900b","Type":"ContainerStarted","Data":"099de6940bed3e0ca6765752c6422b463ea3a5e272d30a9febc0a306baf15a3c"}
Oct 30 23:18:07.320753 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:07.320707    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7" event={"ID":"271aabed-e533-4934-956c-747525e2900b","Type":"ContainerStarted","Data":"1bdcfb50e43e7fc491da38aadb15517fc238a2e927901ee120e057e20f0b3086"}
Oct 30 23:18:07.362005 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:07.361947    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7" podStartSLOduration=2.361929383 podStartE2EDuration="2.361929383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:07.359364303 +0000 UTC m=+4271.398303184" watchObservedRunningTime="2026-10-30 23:18:07.361929383 +0000 UTC m=+4271.400868269"
Oct 30 23:18:08.327872 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:08.327837    4581 generic.go:356] "Generic (PLEG): container finished" podID="271aabed-e533-4934-956c-747525e2900b" containerID="099de6940bed3e0ca6765752c6422b463ea3a5e272d30a9febc0a306baf15a3c" exitCode=0
Oct 30 23:18:08.328297 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:08.327918    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7" event={"ID":"271aabed-e533-4934-956c-747525e2900b","Type":"ContainerDied","Data":"099de6940bed3e0ca6765752c6422b463ea3a5e272d30a9febc0a306baf15a3c"}
Oct 30 23:18:09.489027 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.489002    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:09.568160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.568106    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271aabed-e533-4934-956c-747525e2900b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271aabed-e533-4934-956c-747525e2900b-maas-api-openshift-ca\") pod \"271aabed-e533-4934-956c-747525e2900b\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") "
Oct 30 23:18:09.568414 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.568196    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271aabed-e533-4934-956c-747525e2900b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271aabed-e533-4934-956c-747525e2900b-maas-api-xks-ca\") pod \"271aabed-e533-4934-956c-747525e2900b\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") "
Oct 30 23:18:09.568414 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.568268    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271aabed-e533-4934-956c-747525e2900b-kube-api-access-t7tjc\" (UniqueName: \"kubernetes.io/projected/271aabed-e533-4934-956c-747525e2900b-kube-api-access-t7tjc\") pod \"271aabed-e533-4934-956c-747525e2900b\" (UID: \"271aabed-e533-4934-956c-747525e2900b\") "
Oct 30 23:18:09.568605 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.568574    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271aabed-e533-4934-956c-747525e2900b-maas-api-openshift-ca" pod "271aabed-e533-4934-956c-747525e2900b" (UID: "271aabed-e533-4934-956c-747525e2900b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:18:09.570625 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.570589    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271aabed-e533-4934-956c-747525e2900b-kube-api-access-t7tjc" pod "271aabed-e533-4934-956c-747525e2900b" (UID: "271aabed-e533-4934-956c-747525e2900b"). InnerVolumeSpecName "kube-api-access-t7tjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:18:09.570625 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.570619    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271aabed-e533-4934-956c-747525e2900b-maas-api-xks-ca" pod "271aabed-e533-4934-956c-747525e2900b" (UID: "271aabed-e533-4934-956c-747525e2900b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:09.669819 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.669773    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7tjc\" (UniqueName: \"kubernetes.io/projected/271aabed-e533-4934-956c-747525e2900b-kube-api-access-t7tjc\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:09.669819 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.669806    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271aabed-e533-4934-956c-747525e2900b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:09.669819 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:09.669817    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271aabed-e533-4934-956c-747525e2900b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:10.341044 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:10.341012    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"
Oct 30 23:18:10.341044 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:10.341026    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7" event={"ID":"271aabed-e533-4934-956c-747525e2900b","Type":"ContainerDied","Data":"1bdcfb50e43e7fc491da38aadb15517fc238a2e927901ee120e057e20f0b3086"}
Oct 30 23:18:10.341357 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:10.341070    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bdcfb50e43e7fc491da38aadb15517fc238a2e927901ee120e057e20f0b3086"
Oct 30 23:18:10.816257 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:18:10.816193    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:18:17.855468 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:17.855371    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-24vbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:18:17.889612 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:17.889569    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-24vbj" podUID="9c8d4001-3885-49c0-84b2-1bcb032aa3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:18:24.028178 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:24.028142    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"]
Oct 30 23:18:24.030911 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:24.030879    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-knh5d"]
Oct 30 23:18:24.095411 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:24.095372    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9f4532-6b89-474d-be88-8395dfbdb971" path="/var/lib/kubelet/pods/ee9f4532-6b89-474d-be88-8395dfbdb971/volumes"
Oct 30 23:18:28.521110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.521076    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"]
Oct 30 23:18:28.521612 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.521573    4581 cpu_manager.go:414] "containerMap: removing container" podUID="271aabed-e533-4934-956c-747525e2900b" containerName="revoke-keys"
Oct 30 23:18:28.521612 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.521586    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271aabed-e533-4934-956c-747525e2900b" containerName="revoke-keys"
Oct 30 23:18:28.521706 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.521670    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="271aabed-e533-4934-956c-747525e2900b" containerName="revoke-keys"
Oct 30 23:18:28.525156 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.525135    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.537411 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.537386    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:18:28.547332 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.547302    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:18:28.547615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.547427    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:18:28.547615 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.547530    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:18:28.550993 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.550966    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"]
Oct 30 23:18:28.593577 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.593547    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.593765 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.593630    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.593765 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.593690    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp7md\" (UniqueName: \"kubernetes.io/projected/0e301391-4cba-4b60-8961-7ac668e7b756-kube-api-access-wp7md\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.695304 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.695268    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.695304 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.695309    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp7md\" (UniqueName: \"kubernetes.io/projected/0e301391-4cba-4b60-8961-7ac668e7b756-kube-api-access-wp7md\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.695607 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.695381    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.696002 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.695974    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.698007 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.697988    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.712385 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.712356    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp7md\" (UniqueName: \"kubernetes.io/projected/0e301391-4cba-4b60-8961-7ac668e7b756-kube-api-access-wp7md\") pod \"maas-api-revoke-sub-12e97699-lqd72\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:28.838059 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:28.837973    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:29.141970 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:29.141879    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"]
Oct 30 23:18:29.145713 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:18:29.145682    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e301391_4cba_4b60_8961_7ac668e7b756.slice/crio-adba971f508aeaa2ddbb5308914762ee54d01829adb1ecd715e323ea1c6a44c4 WatchSource:0}: Error finding container adba971f508aeaa2ddbb5308914762ee54d01829adb1ecd715e323ea1c6a44c4: Status 404 returned error can't find the container with id adba971f508aeaa2ddbb5308914762ee54d01829adb1ecd715e323ea1c6a44c4
Oct 30 23:18:29.532354 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:29.532319    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72" event={"ID":"0e301391-4cba-4b60-8961-7ac668e7b756","Type":"ContainerStarted","Data":"341fd6297066917b41bfbbae8a6f5d38fd50fb3119b7dbe3003ce7325d41056c"}
Oct 30 23:18:29.532782 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:29.532377    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72" event={"ID":"0e301391-4cba-4b60-8961-7ac668e7b756","Type":"ContainerStarted","Data":"adba971f508aeaa2ddbb5308914762ee54d01829adb1ecd715e323ea1c6a44c4"}
Oct 30 23:18:29.565744 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:29.565686    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72" podStartSLOduration=1.565673074 podStartE2EDuration="1.565673074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:29.565074663 +0000 UTC m=+4293.604013551" watchObservedRunningTime="2026-10-30 23:18:29.565673074 +0000 UTC m=+4293.604611959"
Oct 30 23:18:30.540342 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:30.540305    4581 generic.go:356] "Generic (PLEG): container finished" podID="0e301391-4cba-4b60-8961-7ac668e7b756" containerID="341fd6297066917b41bfbbae8a6f5d38fd50fb3119b7dbe3003ce7325d41056c" exitCode=0
Oct 30 23:18:30.540752 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:30.540394    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72" event={"ID":"0e301391-4cba-4b60-8961-7ac668e7b756","Type":"ContainerDied","Data":"341fd6297066917b41bfbbae8a6f5d38fd50fb3119b7dbe3003ce7325d41056c"}
Oct 30 23:18:31.681537 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.681509    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:31.725110 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.725073    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e301391-4cba-4b60-8961-7ac668e7b756-kube-api-access-wp7md\" (UniqueName: \"kubernetes.io/projected/0e301391-4cba-4b60-8961-7ac668e7b756-kube-api-access-wp7md\") pod \"0e301391-4cba-4b60-8961-7ac668e7b756\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") "
Oct 30 23:18:31.725319 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.725174    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-openshift-ca\") pod \"0e301391-4cba-4b60-8961-7ac668e7b756\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") "
Oct 30 23:18:31.725319 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.725242    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-xks-ca\") pod \"0e301391-4cba-4b60-8961-7ac668e7b756\" (UID: \"0e301391-4cba-4b60-8961-7ac668e7b756\") "
Oct 30 23:18:31.725552 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.725528    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-openshift-ca" pod "0e301391-4cba-4b60-8961-7ac668e7b756" (UID: "0e301391-4cba-4b60-8961-7ac668e7b756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:18:31.725684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.725667    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:31.727608 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.727578    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e301391-4cba-4b60-8961-7ac668e7b756-kube-api-access-wp7md" pod "0e301391-4cba-4b60-8961-7ac668e7b756" (UID: "0e301391-4cba-4b60-8961-7ac668e7b756"). InnerVolumeSpecName "kube-api-access-wp7md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:18:31.727709 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.727619    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-xks-ca" pod "0e301391-4cba-4b60-8961-7ac668e7b756" (UID: "0e301391-4cba-4b60-8961-7ac668e7b756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:31.827251 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.827131    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e301391-4cba-4b60-8961-7ac668e7b756-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:31.827251 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:31.827163    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp7md\" (UniqueName: \"kubernetes.io/projected/0e301391-4cba-4b60-8961-7ac668e7b756-kube-api-access-wp7md\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:32.553410 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:32.553378    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"
Oct 30 23:18:32.553614 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:32.553380    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72" event={"ID":"0e301391-4cba-4b60-8961-7ac668e7b756","Type":"ContainerDied","Data":"adba971f508aeaa2ddbb5308914762ee54d01829adb1ecd715e323ea1c6a44c4"}
Oct 30 23:18:32.553614 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:32.553511    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adba971f508aeaa2ddbb5308914762ee54d01829adb1ecd715e323ea1c6a44c4"
Oct 30 23:18:40.816294 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:18:40.816257    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:18:41.079001 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.078912    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"]
Oct 30 23:18:41.079402 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.079381    4581 cpu_manager.go:414] "containerMap: removing container" podUID="0e301391-4cba-4b60-8961-7ac668e7b756" containerName="revoke-keys"
Oct 30 23:18:41.079515 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.079405    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e301391-4cba-4b60-8961-7ac668e7b756" containerName="revoke-keys"
Oct 30 23:18:41.079515 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.079487    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e301391-4cba-4b60-8961-7ac668e7b756" containerName="revoke-keys"
Oct 30 23:18:41.082884 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.082866    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.109276 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.109243    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 23:18:41.109492 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.109472    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:18:41.109575 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.109355    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l95g7\""
Oct 30 23:18:41.109575 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.109434    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 23:18:41.109575 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.109282    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:18:41.113105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.113073    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"]
Oct 30 23:18:41.224338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.224306    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-maas-api-tls\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.224338 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.224340    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgz7\" (UniqueName: \"kubernetes.io/projected/e19cf23f-488a-41b6-87cc-4398b5e2126e-kube-api-access-lqgz7\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.224572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.224374    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-metrics-certs\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.224572 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.224530    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19cf23f-488a-41b6-87cc-4398b5e2126e-openshift-service-ca\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.325033 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.324993    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19cf23f-488a-41b6-87cc-4398b5e2126e-openshift-service-ca\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.325264 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.325062    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-maas-api-tls\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.325264 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.325082    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgz7\" (UniqueName: \"kubernetes.io/projected/e19cf23f-488a-41b6-87cc-4398b5e2126e-kube-api-access-lqgz7\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.325264 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.325108    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-metrics-certs\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.325738 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.325717    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19cf23f-488a-41b6-87cc-4398b5e2126e-openshift-service-ca\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.327700 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.327679    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-maas-api-tls\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.327812 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.327731    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-metrics-certs\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.342491 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.342431    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgz7\" (UniqueName: \"kubernetes.io/projected/e19cf23f-488a-41b6-87cc-4398b5e2126e-kube-api-access-lqgz7\") pod \"maas-api-6cf6b4c745-pgxcz\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") " pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.399371 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.399343    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:41.707974 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.707940    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"]
Oct 30 23:18:41.711455 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:18:41.711418    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19cf23f_488a_41b6_87cc_4398b5e2126e.slice/crio-1e9e5398c7497286c5de42889ff3f30c140e1dc036f89d4269bea78d64d8b0f8 WatchSource:0}: Error finding container 1e9e5398c7497286c5de42889ff3f30c140e1dc036f89d4269bea78d64d8b0f8: Status 404 returned error can't find the container with id 1e9e5398c7497286c5de42889ff3f30c140e1dc036f89d4269bea78d64d8b0f8
Oct 30 23:18:41.712727 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:41.712707    4581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:18:42.614066 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:42.614029    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz" event={"ID":"e19cf23f-488a-41b6-87cc-4398b5e2126e","Type":"ContainerStarted","Data":"1e9e5398c7497286c5de42889ff3f30c140e1dc036f89d4269bea78d64d8b0f8"}
Oct 30 23:18:43.621896 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:43.621865    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz" event={"ID":"e19cf23f-488a-41b6-87cc-4398b5e2126e","Type":"ContainerStarted","Data":"4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b"}
Oct 30 23:18:43.622365 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:43.622064    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:43.660459 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:43.660406    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz" podStartSLOduration=1.073125927 podStartE2EDuration="2.66039099s" totalImagesPullingTime="1.587265063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:43.65973329 +0000 UTC m=+4307.698672178" watchObservedRunningTime="2026-10-30 23:18:43.66039099 +0000 UTC m=+4307.699329876"
Oct 30 23:18:48.331249 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.331180    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"]
Oct 30 23:18:48.331687 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.331395    4581 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz" podUID="e19cf23f-488a-41b6-87cc-4398b5e2126e" containerName="maas-api" containerID="cri-o://4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b" gracePeriod=30
Oct 30 23:18:48.337591 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.337569    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:48.594414 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.594382    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:48.654040 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.654004    4581 generic.go:356] "Generic (PLEG): container finished" podID="e19cf23f-488a-41b6-87cc-4398b5e2126e" containerID="4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b" exitCode=0
Oct 30 23:18:48.654264 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.654063    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz" event={"ID":"e19cf23f-488a-41b6-87cc-4398b5e2126e","Type":"ContainerDied","Data":"4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b"}
Oct 30 23:18:48.654264 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.654103    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"
Oct 30 23:18:48.654264 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.654119    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz" event={"ID":"e19cf23f-488a-41b6-87cc-4398b5e2126e","Type":"ContainerDied","Data":"1e9e5398c7497286c5de42889ff3f30c140e1dc036f89d4269bea78d64d8b0f8"}
Oct 30 23:18:48.654264 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.654150    4581 scope.go:122] "RemoveContainer" containerID="4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b"
Oct 30 23:18:48.664850 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.664824    4581 scope.go:122] "RemoveContainer" containerID="4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b"
Oct 30 23:18:48.665189 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:18:48.665165    4581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b\": container with ID starting with 4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b not found: ID does not exist" containerID="4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b"
Oct 30 23:18:48.665302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.665207    4581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b"} err="failed to get container status \"4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b\": rpc error: code = NotFound desc = could not find container \"4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b\": container with ID starting with 4c533302be11faa0f2de6cd9c4614c8779ac372368a6f4b7fe8476755ee3f74b not found: ID does not exist"
Oct 30 23:18:48.702586 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.702556    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-maas-api-tls\") pod \"e19cf23f-488a-41b6-87cc-4398b5e2126e\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") "
Oct 30 23:18:48.702796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.702645    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19cf23f-488a-41b6-87cc-4398b5e2126e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19cf23f-488a-41b6-87cc-4398b5e2126e-openshift-service-ca\") pod \"e19cf23f-488a-41b6-87cc-4398b5e2126e\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") "
Oct 30 23:18:48.702796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.702689    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19cf23f-488a-41b6-87cc-4398b5e2126e-kube-api-access-lqgz7\" (UniqueName: \"kubernetes.io/projected/e19cf23f-488a-41b6-87cc-4398b5e2126e-kube-api-access-lqgz7\") pod \"e19cf23f-488a-41b6-87cc-4398b5e2126e\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") "
Oct 30 23:18:48.702796 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.702765    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-metrics-certs\") pod \"e19cf23f-488a-41b6-87cc-4398b5e2126e\" (UID: \"e19cf23f-488a-41b6-87cc-4398b5e2126e\") "
Oct 30 23:18:48.703057 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.703025    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19cf23f-488a-41b6-87cc-4398b5e2126e-openshift-service-ca" pod "e19cf23f-488a-41b6-87cc-4398b5e2126e" (UID: "e19cf23f-488a-41b6-87cc-4398b5e2126e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:18:48.703162 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.703142    4581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19cf23f-488a-41b6-87cc-4398b5e2126e-openshift-service-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:48.705017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.704988    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-metrics-certs" pod "e19cf23f-488a-41b6-87cc-4398b5e2126e" (UID: "e19cf23f-488a-41b6-87cc-4398b5e2126e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:48.705109 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.705039    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-maas-api-tls" pod "e19cf23f-488a-41b6-87cc-4398b5e2126e" (UID: "e19cf23f-488a-41b6-87cc-4398b5e2126e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:48.705155 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.705099    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19cf23f-488a-41b6-87cc-4398b5e2126e-kube-api-access-lqgz7" pod "e19cf23f-488a-41b6-87cc-4398b5e2126e" (UID: "e19cf23f-488a-41b6-87cc-4398b5e2126e"). InnerVolumeSpecName "kube-api-access-lqgz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:18:48.804364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.804326    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgz7\" (UniqueName: \"kubernetes.io/projected/e19cf23f-488a-41b6-87cc-4398b5e2126e-kube-api-access-lqgz7\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:48.804364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.804357    4581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-metrics-certs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:48.804364 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.804366    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19cf23f-488a-41b6-87cc-4398b5e2126e-maas-api-tls\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:18:48.993926 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.993892    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"]
Oct 30 23:18:48.997487 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:48.997462    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf6b4c745-pgxcz"]
Oct 30 23:18:50.096956 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:18:50.096918    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19cf23f-488a-41b6-87cc-4398b5e2126e" path="/var/lib/kubelet/pods/e19cf23f-488a-41b6-87cc-4398b5e2126e/volumes"
Oct 30 23:19:06.616011 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.615974    4581 scope.go:122] "RemoveContainer" containerID="b16619bb7c7ebb7c1693f4718ec5968e242f103c2c59979da252764e61a9a3d4"
Oct 30 23:19:06.625379 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.625356    4581 scope.go:122] "RemoveContainer" containerID="068522a9bb728515539791102816303625ddb2075afb20b8f1bbf78f9021df26"
Oct 30 23:19:06.834519 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.834482    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"]
Oct 30 23:19:06.834911 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.834896    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e19cf23f-488a-41b6-87cc-4398b5e2126e" containerName="maas-api"
Oct 30 23:19:06.835017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.834911    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19cf23f-488a-41b6-87cc-4398b5e2126e" containerName="maas-api"
Oct 30 23:19:06.835017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.835009    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19cf23f-488a-41b6-87cc-4398b5e2126e" containerName="maas-api"
Oct 30 23:19:06.839628 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.839610    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:06.851609 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.851583    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:19:06.851868 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.851849    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:19:06.851990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.851882    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:19:06.861480 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.861453    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:19:06.865086 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.865065    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"]
Oct 30 23:19:06.881160 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.881095    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:06.881304 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.881159    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:06.881385 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.881314    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p694c\" (UniqueName: \"kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:06.981926 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.981887    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:06.982108 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.981950    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:06.982108 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.981992    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p694c\" (UniqueName: \"kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:06.982733 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.982705    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:06.984546 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:06.984515    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:07.004229 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:19:07.004192    4581 projected.go:196] Error preparing data for projected volume kube-api-access-p694c for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 23:19:07.004369 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:19:07.004284    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c podName:22aead2f-caad-4963-862e-9c8e3cde59a0 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:07.504269087 +0000 UTC m=+4331.543207964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p694c" (UniqueName: "kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c") pod "maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" (UID: "22aead2f-caad-4963-862e-9c8e3cde59a0") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 23:19:07.587401 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:07.587356    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p694c\" (UniqueName: \"kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:07.599548 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:19:07.599518    4581 projected.go:196] Error preparing data for projected volume kube-api-access-p694c for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 23:19:07.599712 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:19:07.599593    4581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c podName:22aead2f-caad-4963-862e-9c8e3cde59a0 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:08.599573378 +0000 UTC m=+4332.638512258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p694c" (UniqueName: "kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c") pod "maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" (UID: "22aead2f-caad-4963-862e-9c8e3cde59a0") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 23:19:08.696979 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:08.696938    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p694c\" (UniqueName: \"kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:08.713684 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:08.713657    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p694c\" (UniqueName: \"kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c\") pod \"maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:08.952017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:08.951935    4581 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" secret="" err="secret \"maas-api-cleanup-dockercfg-vp5mn\" not found"
Oct 30 23:19:08.952017 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:08.951989    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:09.253521 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:09.253436    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"]
Oct 30 23:19:09.257305 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:19:09.257266    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22aead2f_caad_4963_862e_9c8e3cde59a0.slice/crio-1b993594f36809b66c577a8fc080db3d943cd31504711384118513bf5fc39c61 WatchSource:0}: Error finding container 1b993594f36809b66c577a8fc080db3d943cd31504711384118513bf5fc39c61: Status 404 returned error can't find the container with id 1b993594f36809b66c577a8fc080db3d943cd31504711384118513bf5fc39c61
Oct 30 23:19:09.786679 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:09.786636    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" event={"ID":"22aead2f-caad-4963-862e-9c8e3cde59a0","Type":"ContainerStarted","Data":"01ed39fb9bbae86a3274ee494fda1f9e111d78d9749a37666e97c8e7c5129eb1"}
Oct 30 23:19:09.787279 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:09.786697    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" event={"ID":"22aead2f-caad-4963-862e-9c8e3cde59a0","Type":"ContainerStarted","Data":"1b993594f36809b66c577a8fc080db3d943cd31504711384118513bf5fc39c61"}
Oct 30 23:19:09.787279 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:09.786922    4581 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" secret="" err="secret \"maas-api-cleanup-dockercfg-vp5mn\" not found"
Oct 30 23:19:09.839888 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:09.839778    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" podStartSLOduration=3.839765048 podStartE2EDuration="3.839765048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:09.838555259 +0000 UTC m=+4333.877494146" watchObservedRunningTime="2026-10-30 23:19:09.839765048 +0000 UTC m=+4333.878703912"
Oct 30 23:19:10.794626 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:10.794591    4581 generic.go:356] "Generic (PLEG): container finished" podID="22aead2f-caad-4963-862e-9c8e3cde59a0" containerID="01ed39fb9bbae86a3274ee494fda1f9e111d78d9749a37666e97c8e7c5129eb1" exitCode=0
Oct 30 23:19:10.795122 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:10.794676    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" event={"ID":"22aead2f-caad-4963-862e-9c8e3cde59a0","Type":"ContainerDied","Data":"01ed39fb9bbae86a3274ee494fda1f9e111d78d9749a37666e97c8e7c5129eb1"}
Oct 30 23:19:10.816671 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:19:10.816635    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:19:11.923790 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:11.923763    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:12.028564 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.028522    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c\" (UniqueName: \"kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c\") pod \"22aead2f-caad-4963-862e-9c8e3cde59a0\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") "
Oct 30 23:19:12.028756 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.028609    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-openshift-ca\") pod \"22aead2f-caad-4963-862e-9c8e3cde59a0\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") "
Oct 30 23:19:12.028756 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.028648    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-xks-ca\") pod \"22aead2f-caad-4963-862e-9c8e3cde59a0\" (UID: \"22aead2f-caad-4963-862e-9c8e3cde59a0\") "
Oct 30 23:19:12.029001 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.028975    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-openshift-ca" pod "22aead2f-caad-4963-862e-9c8e3cde59a0" (UID: "22aead2f-caad-4963-862e-9c8e3cde59a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:12.030977 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.030950    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-xks-ca" pod "22aead2f-caad-4963-862e-9c8e3cde59a0" (UID: "22aead2f-caad-4963-862e-9c8e3cde59a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:19:12.030977 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.030959    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c" pod "22aead2f-caad-4963-862e-9c8e3cde59a0" (UID: "22aead2f-caad-4963-862e-9c8e3cde59a0"). InnerVolumeSpecName "kube-api-access-p694c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:12.130788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.130572    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:19:12.130788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.130606    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22aead2f-caad-4963-862e-9c8e3cde59a0-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:19:12.130788 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.130621    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p694c\" (UniqueName: \"kubernetes.io/projected/22aead2f-caad-4963-862e-9c8e3cde59a0-kube-api-access-p694c\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:19:12.808130 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.808100    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"
Oct 30 23:19:12.808374 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.808126    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7" event={"ID":"22aead2f-caad-4963-862e-9c8e3cde59a0","Type":"ContainerDied","Data":"1b993594f36809b66c577a8fc080db3d943cd31504711384118513bf5fc39c61"}
Oct 30 23:19:12.808374 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:12.808174    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b993594f36809b66c577a8fc080db3d943cd31504711384118513bf5fc39c61"
Oct 30 23:19:20.225334 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:20.225296    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"]
Oct 30 23:19:20.227827 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:20.227794    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1595eccb-wzqc7"]
Oct 30 23:19:22.096230 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:19:22.096179    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22aead2f-caad-4963-862e-9c8e3cde59a0" path="/var/lib/kubelet/pods/22aead2f-caad-4963-862e-9c8e3cde59a0/volumes"
Oct 30 23:19:40.815601 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:19:40.815512    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:20:06.655748 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:20:06.655714    4581 scope.go:122] "RemoveContainer" containerID="f572d14934e64e6ec9b3c43f9a9babb1a5680fb4104cdf1d28c812f915c7a0fe"
Oct 30 23:20:10.816098 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:20:10.816062    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:20:29.027766 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:20:29.027714    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"]
Oct 30 23:20:29.036942 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:20:29.036912    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-rfsw9"]
Oct 30 23:20:30.094716 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:20:30.094670    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abb5dd5-080a-4fd1-aec7-67284dc604ac" path="/var/lib/kubelet/pods/6abb5dd5-080a-4fd1-aec7-67284dc604ac/volumes"
Oct 30 23:20:40.816390 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:20:40.816349    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:21:03.971101 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:03.971061    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"]
Oct 30 23:21:03.971614 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:03.971489    4581 cpu_manager.go:414] "containerMap: removing container" podUID="22aead2f-caad-4963-862e-9c8e3cde59a0" containerName="revoke-keys"
Oct 30 23:21:03.971614 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:03.971500    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22aead2f-caad-4963-862e-9c8e3cde59a0" containerName="revoke-keys"
Oct 30 23:21:03.971614 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:03.971575    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="22aead2f-caad-4963-862e-9c8e3cde59a0" containerName="revoke-keys"
Oct 30 23:21:03.975100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:03.974819    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:03.987536 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:03.987515    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:21:03.997027 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:03.997003    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:21:04.000782 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.000756    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"]
Oct 30 23:21:04.061462 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.061429    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:21:04.061639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.061491    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9rj8q\""
Oct 30 23:21:04.145142 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.145103    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.145390 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.145292    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.145390 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.145349    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhpn\" (UniqueName: \"kubernetes.io/projected/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-kube-api-access-5zhpn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.246559 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.246458    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.246559 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.246523    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhpn\" (UniqueName: \"kubernetes.io/projected/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-kube-api-access-5zhpn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.246819 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.246641    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.247326 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.247302    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.249097 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.249079    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.263466 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.263438    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhpn\" (UniqueName: \"kubernetes.io/projected/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-kube-api-access-5zhpn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.292302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.292274    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:04.595778 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:04.595698    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"]
Oct 30 23:21:04.598827 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:21:04.598796    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd38a52_258f_4a4e_95b6_2caf85f1d8d4.slice/crio-3eb154ae6b3ff6fa744e7d20300a5fa3e0d8c6581bf88ce741f8c41646bbe667 WatchSource:0}: Error finding container 3eb154ae6b3ff6fa744e7d20300a5fa3e0d8c6581bf88ce741f8c41646bbe667: Status 404 returned error can't find the container with id 3eb154ae6b3ff6fa744e7d20300a5fa3e0d8c6581bf88ce741f8c41646bbe667
Oct 30 23:21:05.487173 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:05.487136    4581 generic.go:356] "Generic (PLEG): container finished" podID="6fd38a52-258f-4a4e-95b6-2caf85f1d8d4" containerID="67a072210d97a748de594ffed966c6f9e6f09b3ce86a30c54db5c3f1050aa7ad" exitCode=0
Oct 30 23:21:05.487644 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:05.487175    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6" event={"ID":"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4","Type":"ContainerDied","Data":"67a072210d97a748de594ffed966c6f9e6f09b3ce86a30c54db5c3f1050aa7ad"}
Oct 30 23:21:05.487644 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:05.487293    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6" event={"ID":"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4","Type":"ContainerStarted","Data":"3eb154ae6b3ff6fa744e7d20300a5fa3e0d8c6581bf88ce741f8c41646bbe667"}
Oct 30 23:21:06.625344 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.625315    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:06.773141 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.773064    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-kube-api-access-5zhpn\" (UniqueName: \"kubernetes.io/projected/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-kube-api-access-5zhpn\") pod \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") "
Oct 30 23:21:06.773141 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.773105    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-openshift-ca\") pod \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") "
Oct 30 23:21:06.773395 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.773232    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-xks-ca\") pod \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\" (UID: \"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4\") "
Oct 30 23:21:06.773639 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.773609    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-openshift-ca" pod "6fd38a52-258f-4a4e-95b6-2caf85f1d8d4" (UID: "6fd38a52-258f-4a4e-95b6-2caf85f1d8d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:21:06.775356 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.775323    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-kube-api-access-5zhpn" pod "6fd38a52-258f-4a4e-95b6-2caf85f1d8d4" (UID: "6fd38a52-258f-4a4e-95b6-2caf85f1d8d4"). InnerVolumeSpecName "kube-api-access-5zhpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:21:06.775498 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.775414    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-xks-ca" pod "6fd38a52-258f-4a4e-95b6-2caf85f1d8d4" (UID: "6fd38a52-258f-4a4e-95b6-2caf85f1d8d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:21:06.874992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.874955    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:21:06.874992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.874987    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhpn\" (UniqueName: \"kubernetes.io/projected/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-kube-api-access-5zhpn\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:21:06.874992 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:06.874998    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:21:07.500054 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:07.500025    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"
Oct 30 23:21:07.500260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:07.500072    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6" event={"ID":"6fd38a52-258f-4a4e-95b6-2caf85f1d8d4","Type":"ContainerDied","Data":"3eb154ae6b3ff6fa744e7d20300a5fa3e0d8c6581bf88ce741f8c41646bbe667"}
Oct 30 23:21:07.500260 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:07.500121    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb154ae6b3ff6fa744e7d20300a5fa3e0d8c6581bf88ce741f8c41646bbe667"
Oct 30 23:21:10.817037 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:21:10.817001    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:21:21.094905 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:21.094873    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"]
Oct 30 23:21:21.103542 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:21.103501    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9cdd18-xfbd6"]
Oct 30 23:21:22.094900 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:22.094852    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd38a52-258f-4a4e-95b6-2caf85f1d8d4" path="/var/lib/kubelet/pods/6fd38a52-258f-4a4e-95b6-2caf85f1d8d4/volumes"
Oct 30 23:21:36.027683 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:36.027644    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"]
Oct 30 23:21:36.037046 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:36.037006    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1b3c-9nn7m"]
Oct 30 23:21:36.095007 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:36.094971    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56acd89d-7426-4525-8813-2384c2220559" path="/var/lib/kubelet/pods/56acd89d-7426-4525-8813-2384c2220559/volumes"
Oct 30 23:21:40.967771 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:21:40.967694    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:21:42.222503 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:42.222457    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:21:42.223336 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:42.223302    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:21:42.254245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:42.250367    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:21:42.254245 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:42.250441    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:21:45.360491 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.360457    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"]
Oct 30 23:21:45.360888 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.360854    4581 cpu_manager.go:414] "containerMap: removing container" podUID="6fd38a52-258f-4a4e-95b6-2caf85f1d8d4" containerName="revoke-keys"
Oct 30 23:21:45.360888 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.360865    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd38a52-258f-4a4e-95b6-2caf85f1d8d4" containerName="revoke-keys"
Oct 30 23:21:45.360964 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.360942    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd38a52-258f-4a4e-95b6-2caf85f1d8d4" containerName="revoke-keys"
Oct 30 23:21:45.364302 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.364286    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.376447 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.376424    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:21:45.386199 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.386165    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:21:45.386374 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.386201    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9rj8q\""
Oct 30 23:21:45.386374 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.386257    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:21:45.388937 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.388918    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"]
Oct 30 23:21:45.525711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.525681    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjsst\" (UniqueName: \"kubernetes.io/projected/3b9024ab-69ce-4324-b364-61cc3f20745b-kube-api-access-cjsst\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.525711 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.525714    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.525942 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.525886    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.626698 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.626610    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.626698 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.626673    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjsst\" (UniqueName: \"kubernetes.io/projected/3b9024ab-69ce-4324-b364-61cc3f20745b-kube-api-access-cjsst\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.626896 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.626701    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.627488 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.627462    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.629353 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.629322    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.644668 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.644638    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjsst\" (UniqueName: \"kubernetes.io/projected/3b9024ab-69ce-4324-b364-61cc3f20745b-kube-api-access-cjsst\") pod \"maas-api-revoke-sub-f03629db-kh565\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.679874 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.679839    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:45.979507 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:45.979419    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"]
Oct 30 23:21:45.982938 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:21:45.982908    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9024ab_69ce_4324_b364_61cc3f20745b.slice/crio-e090adf3cb7d96133cb9292e17c0b68d7497b5b902b4b1543bdbe08ea1ffcb40 WatchSource:0}: Error finding container e090adf3cb7d96133cb9292e17c0b68d7497b5b902b4b1543bdbe08ea1ffcb40: Status 404 returned error can't find the container with id e090adf3cb7d96133cb9292e17c0b68d7497b5b902b4b1543bdbe08ea1ffcb40
Oct 30 23:21:46.797726 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:46.797696    4581 generic.go:356] "Generic (PLEG): container finished" podID="3b9024ab-69ce-4324-b364-61cc3f20745b" containerID="009e99471a38141c64b30cee053ed5155a8e929214ca24753dd2099ca3359e71" exitCode=0
Oct 30 23:21:46.798114 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:46.797769    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565" event={"ID":"3b9024ab-69ce-4324-b364-61cc3f20745b","Type":"ContainerDied","Data":"009e99471a38141c64b30cee053ed5155a8e929214ca24753dd2099ca3359e71"}
Oct 30 23:21:46.798114 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:46.797806    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565" event={"ID":"3b9024ab-69ce-4324-b364-61cc3f20745b","Type":"ContainerStarted","Data":"e090adf3cb7d96133cb9292e17c0b68d7497b5b902b4b1543bdbe08ea1ffcb40"}
Oct 30 23:21:47.948105 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:47.948076    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:48.048660 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.048624    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-openshift-ca\") pod \"3b9024ab-69ce-4324-b364-61cc3f20745b\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") "
Oct 30 23:21:48.048871 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.048685    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-xks-ca\") pod \"3b9024ab-69ce-4324-b364-61cc3f20745b\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") "
Oct 30 23:21:48.048871 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.048762    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9024ab-69ce-4324-b364-61cc3f20745b-kube-api-access-cjsst\" (UniqueName: \"kubernetes.io/projected/3b9024ab-69ce-4324-b364-61cc3f20745b-kube-api-access-cjsst\") pod \"3b9024ab-69ce-4324-b364-61cc3f20745b\" (UID: \"3b9024ab-69ce-4324-b364-61cc3f20745b\") "
Oct 30 23:21:48.049071 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.049047    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-openshift-ca" pod "3b9024ab-69ce-4324-b364-61cc3f20745b" (UID: "3b9024ab-69ce-4324-b364-61cc3f20745b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:21:48.051034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.051011    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-xks-ca" pod "3b9024ab-69ce-4324-b364-61cc3f20745b" (UID: "3b9024ab-69ce-4324-b364-61cc3f20745b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:21:48.051127 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.051103    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9024ab-69ce-4324-b364-61cc3f20745b-kube-api-access-cjsst" pod "3b9024ab-69ce-4324-b364-61cc3f20745b" (UID: "3b9024ab-69ce-4324-b364-61cc3f20745b"). InnerVolumeSpecName "kube-api-access-cjsst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:21:48.149769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.149723    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:21:48.149769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.149768    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9024ab-69ce-4324-b364-61cc3f20745b-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:21:48.150001 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.149783    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjsst\" (UniqueName: \"kubernetes.io/projected/3b9024ab-69ce-4324-b364-61cc3f20745b-kube-api-access-cjsst\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:21:48.810404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.810374    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565"
Oct 30 23:21:48.810404 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.810381    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kh565" event={"ID":"3b9024ab-69ce-4324-b364-61cc3f20745b","Type":"ContainerDied","Data":"e090adf3cb7d96133cb9292e17c0b68d7497b5b902b4b1543bdbe08ea1ffcb40"}
Oct 30 23:21:48.810625 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:48.810431    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e090adf3cb7d96133cb9292e17c0b68d7497b5b902b4b1543bdbe08ea1ffcb40"
Oct 30 23:21:57.691593 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:57.691559    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:21:57.701183 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:21:57.701159    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqbrc_ad9995b8-ba05-47ba-81c6-6f74c234f2c1/ovn-acl-logging/0.log"
Oct 30 23:22:02.516426 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.516393    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"]
Oct 30 23:22:02.516841 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.516793    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3b9024ab-69ce-4324-b364-61cc3f20745b" containerName="revoke-keys"
Oct 30 23:22:02.516841 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.516803    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9024ab-69ce-4324-b364-61cc3f20745b" containerName="revoke-keys"
Oct 30 23:22:02.516926 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.516882    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9024ab-69ce-4324-b364-61cc3f20745b" containerName="revoke-keys"
Oct 30 23:22:02.519922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.519905    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.532076 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.532030    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:22:02.541783 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.541740    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:22:02.541957 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.541856    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9rj8q\""
Oct 30 23:22:02.542118 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.542071    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:22:02.545396 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.545368    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"]
Oct 30 23:22:02.633421 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.633379    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.633587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.633555    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.633587 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.633582    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2jhs\" (UniqueName: \"kubernetes.io/projected/e7b6f703-1bff-4312-b59f-0d9188b00754-kube-api-access-z2jhs\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.734449 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.734410    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.734646 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.734522    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.734646 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.734545    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2jhs\" (UniqueName: \"kubernetes.io/projected/e7b6f703-1bff-4312-b59f-0d9188b00754-kube-api-access-z2jhs\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.735388 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.735284    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.737003 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.736978    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.751939 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.751910    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2jhs\" (UniqueName: \"kubernetes.io/projected/e7b6f703-1bff-4312-b59f-0d9188b00754-kube-api-access-z2jhs\") pod \"maas-api-revoke-sub-0165d7d2-fb84f\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:02.833721 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:02.833634    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:03.135077 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:03.134888    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"]
Oct 30 23:22:03.138076 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:22:03.138042    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b6f703_1bff_4312_b59f_0d9188b00754.slice/crio-d0d36146b3910d44312a19ca3133ca015431145e23704bd24c6c535896d9195b WatchSource:0}: Error finding container d0d36146b3910d44312a19ca3133ca015431145e23704bd24c6c535896d9195b: Status 404 returned error can't find the container with id d0d36146b3910d44312a19ca3133ca015431145e23704bd24c6c535896d9195b
Oct 30 23:22:03.924114 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:03.924071    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f" event={"ID":"e7b6f703-1bff-4312-b59f-0d9188b00754","Type":"ContainerStarted","Data":"eb61ce24c223ce9f2780433b00cad77ae9522b987631296582f3f0d2c0651380"}
Oct 30 23:22:03.924553 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:03.924137    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f" event={"ID":"e7b6f703-1bff-4312-b59f-0d9188b00754","Type":"ContainerStarted","Data":"d0d36146b3910d44312a19ca3133ca015431145e23704bd24c6c535896d9195b"}
Oct 30 23:22:03.974002 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:03.973941    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f" podStartSLOduration=1.973921539 podStartE2EDuration="1.973921539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:03.970391031 +0000 UTC m=+4508.009329919" watchObservedRunningTime="2026-10-30 23:22:03.973921539 +0000 UTC m=+4508.012860427"
Oct 30 23:22:04.931075 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:04.931042    4581 generic.go:356] "Generic (PLEG): container finished" podID="e7b6f703-1bff-4312-b59f-0d9188b00754" containerID="eb61ce24c223ce9f2780433b00cad77ae9522b987631296582f3f0d2c0651380" exitCode=0
Oct 30 23:22:04.931530 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:04.931127    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f" event={"ID":"e7b6f703-1bff-4312-b59f-0d9188b00754","Type":"ContainerDied","Data":"eb61ce24c223ce9f2780433b00cad77ae9522b987631296582f3f0d2c0651380"}
Oct 30 23:22:06.074442 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.074414    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:06.168643 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.168612    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b6f703-1bff-4312-b59f-0d9188b00754-kube-api-access-z2jhs\" (UniqueName: \"kubernetes.io/projected/e7b6f703-1bff-4312-b59f-0d9188b00754-kube-api-access-z2jhs\") pod \"e7b6f703-1bff-4312-b59f-0d9188b00754\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") "
Oct 30 23:22:06.168854 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.168669    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-xks-ca\") pod \"e7b6f703-1bff-4312-b59f-0d9188b00754\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") "
Oct 30 23:22:06.168854 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.168788    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-openshift-ca\") pod \"e7b6f703-1bff-4312-b59f-0d9188b00754\" (UID: \"e7b6f703-1bff-4312-b59f-0d9188b00754\") "
Oct 30 23:22:06.169187 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.169162    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-openshift-ca" pod "e7b6f703-1bff-4312-b59f-0d9188b00754" (UID: "e7b6f703-1bff-4312-b59f-0d9188b00754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:22:06.170791 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.170769    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-xks-ca" pod "e7b6f703-1bff-4312-b59f-0d9188b00754" (UID: "e7b6f703-1bff-4312-b59f-0d9188b00754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:22:06.170930 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.170913    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b6f703-1bff-4312-b59f-0d9188b00754-kube-api-access-z2jhs" pod "e7b6f703-1bff-4312-b59f-0d9188b00754" (UID: "e7b6f703-1bff-4312-b59f-0d9188b00754"). InnerVolumeSpecName "kube-api-access-z2jhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:06.270233 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.270138    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:22:06.270233 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.270168    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2jhs\" (UniqueName: \"kubernetes.io/projected/e7b6f703-1bff-4312-b59f-0d9188b00754-kube-api-access-z2jhs\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:22:06.270233 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.270180    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6f703-1bff-4312-b59f-0d9188b00754-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:22:06.689103 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.689066    4581 scope.go:122] "RemoveContainer" containerID="51f8545367791fae63223087a5eebedd6df211ec4e1d4074e92190c7044cc1b0"
Oct 30 23:22:06.944569 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.944485    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f" event={"ID":"e7b6f703-1bff-4312-b59f-0d9188b00754","Type":"ContainerDied","Data":"d0d36146b3910d44312a19ca3133ca015431145e23704bd24c6c535896d9195b"}
Oct 30 23:22:06.944569 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.944527    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fb84f"
Oct 30 23:22:06.944569 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:06.944543    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0d36146b3910d44312a19ca3133ca015431145e23704bd24c6c535896d9195b"
Oct 30 23:22:10.816275 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:22:10.816239    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:22:13.478922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.478811    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"]
Oct 30 23:22:13.479662 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.479301    4581 cpu_manager.go:414] "containerMap: removing container" podUID="e7b6f703-1bff-4312-b59f-0d9188b00754" containerName="revoke-keys"
Oct 30 23:22:13.479662 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.479317    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b6f703-1bff-4312-b59f-0d9188b00754" containerName="revoke-keys"
Oct 30 23:22:13.479662 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.479405    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b6f703-1bff-4312-b59f-0d9188b00754" containerName="revoke-keys"
Oct 30 23:22:13.484016 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.483994    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.505731 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.505695    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:22:13.505731 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.505716    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9rj8q\""
Oct 30 23:22:13.505958 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.505695    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:22:13.505958 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.505810    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:22:13.508859 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.508832    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"]
Oct 30 23:22:13.642093 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.642036    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.642343 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.642260    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.642343 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.642312    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkpjw\" (UniqueName: \"kubernetes.io/projected/3e395fdb-367e-47d9-ae73-bc7d973dfc76-kube-api-access-nkpjw\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.743562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.743475    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.743562 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.743535    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkpjw\" (UniqueName: \"kubernetes.io/projected/3e395fdb-367e-47d9-ae73-bc7d973dfc76-kube-api-access-nkpjw\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.743786 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.743577    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.744360 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.744330    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.746046 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.746027    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.760500 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.760475    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkpjw\" (UniqueName: \"kubernetes.io/projected/3e395fdb-367e-47d9-ae73-bc7d973dfc76-kube-api-access-nkpjw\") pod \"maas-api-revoke-sub-6b19338f-pz46x\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:13.801457 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:13.801429    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:14.108151 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:14.108119    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"]
Oct 30 23:22:14.115289 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:22:14.115259    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e395fdb_367e_47d9_ae73_bc7d973dfc76.slice/crio-8602905b5af585ad5d26be9d5518cdf89471c44d99a320ce9f8df5945e118ba1 WatchSource:0}: Error finding container 8602905b5af585ad5d26be9d5518cdf89471c44d99a320ce9f8df5945e118ba1: Status 404 returned error can't find the container with id 8602905b5af585ad5d26be9d5518cdf89471c44d99a320ce9f8df5945e118ba1
Oct 30 23:22:14.990538 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:14.990508    4581 generic.go:356] "Generic (PLEG): container finished" podID="3e395fdb-367e-47d9-ae73-bc7d973dfc76" containerID="1dd65cdfe044a88f3dd5559dd368964259ea09e9c6ae2174616886ef23839a57" exitCode=0
Oct 30 23:22:14.990999 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:14.990551    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x" event={"ID":"3e395fdb-367e-47d9-ae73-bc7d973dfc76","Type":"ContainerDied","Data":"1dd65cdfe044a88f3dd5559dd368964259ea09e9c6ae2174616886ef23839a57"}
Oct 30 23:22:14.990999 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:14.990602    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x" event={"ID":"3e395fdb-367e-47d9-ae73-bc7d973dfc76","Type":"ContainerStarted","Data":"8602905b5af585ad5d26be9d5518cdf89471c44d99a320ce9f8df5945e118ba1"}
Oct 30 23:22:16.137316 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.137286    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:16.271545 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.271458    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e395fdb-367e-47d9-ae73-bc7d973dfc76-kube-api-access-nkpjw\" (UniqueName: \"kubernetes.io/projected/3e395fdb-367e-47d9-ae73-bc7d973dfc76-kube-api-access-nkpjw\") pod \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") "
Oct 30 23:22:16.271545 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.271514    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-openshift-ca\") pod \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") "
Oct 30 23:22:16.271794 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.271548    4581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-xks-ca\") pod \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\" (UID: \"3e395fdb-367e-47d9-ae73-bc7d973dfc76\") "
Oct 30 23:22:16.271897 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.271868    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-openshift-ca" pod "3e395fdb-367e-47d9-ae73-bc7d973dfc76" (UID: "3e395fdb-367e-47d9-ae73-bc7d973dfc76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:22:16.273717 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.273690    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-xks-ca" pod "3e395fdb-367e-47d9-ae73-bc7d973dfc76" (UID: "3e395fdb-367e-47d9-ae73-bc7d973dfc76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:22:16.273851 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.273827    4581 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e395fdb-367e-47d9-ae73-bc7d973dfc76-kube-api-access-nkpjw" pod "3e395fdb-367e-47d9-ae73-bc7d973dfc76" (UID: "3e395fdb-367e-47d9-ae73-bc7d973dfc76"). InnerVolumeSpecName "kube-api-access-nkpjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:16.372811 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.372773    4581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkpjw\" (UniqueName: \"kubernetes.io/projected/3e395fdb-367e-47d9-ae73-bc7d973dfc76-kube-api-access-nkpjw\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:22:16.372811 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.372805    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-openshift-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:22:16.372811 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:16.372816    4581 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e395fdb-367e-47d9-ae73-bc7d973dfc76-maas-api-xks-ca\") on node \"ip-10-0-135-132.ec2.internal\" DevicePath \"\""
Oct 30 23:22:17.003092 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:17.003060    4581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x"
Oct 30 23:22:17.003322 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:17.003065    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pz46x" event={"ID":"3e395fdb-367e-47d9-ae73-bc7d973dfc76","Type":"ContainerDied","Data":"8602905b5af585ad5d26be9d5518cdf89471c44d99a320ce9f8df5945e118ba1"}
Oct 30 23:22:17.003322 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:17.003179    4581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8602905b5af585ad5d26be9d5518cdf89471c44d99a320ce9f8df5945e118ba1"
Oct 30 23:22:25.027782 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:25.027744    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd789d65d-jvcf8_cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec/authorino/0.log"
Oct 30 23:22:27.257247 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:27.253429    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tbk5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:22:27.257247 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:27.253502    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tbk5w" podUID="e50d3c0f-8e93-4e9b-9983-761f99918584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:22:27.257247 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:27.254032    4581 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h8mv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:22:27.257247 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:27.254095    4581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h8mv5" podUID="20b2d85e-38be-49c2-8ef6-31644a816dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:22:35.575947 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:35.575910    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d5c88fbff-mvdht_9e4af176-ced2-4a38-93f8-cbe8130eaef2/manager/0.log"
Oct 30 23:22:38.312843 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:38.312762    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fb84f_e7b6f703-1bff-4312-b59f-0d9188b00754/revoke-keys/0.log"
Oct 30 23:22:38.497687 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:38.497660    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-12e97699-lqd72_0e301391-4cba-4b60-8961-7ac668e7b756/revoke-keys/0.log"
Oct 30 23:22:38.686915 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:38.686879    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-45gj7_271aabed-e533-4934-956c-747525e2900b/revoke-keys/0.log"
Oct 30 23:22:38.882617 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:38.882591    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-sqlpq_66ede7f6-f043-4dee-9191-7492fe5bf7e3/revoke-keys/0.log"
Oct 30 23:22:39.077367 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:39.077287    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-pz46x_3e395fdb-367e-47d9-ae73-bc7d973dfc76/revoke-keys/0.log"
Oct 30 23:22:39.271595 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:39.271566    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-kh565_3b9024ab-69ce-4324-b364-61cc3f20745b/revoke-keys/0.log"
Oct 30 23:22:40.816586 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:22:40.816550    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:22:40.920062 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:40.920034    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd789d65d-jvcf8_cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec/authorino/0.log"
Oct 30 23:22:41.163990 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:41.163965    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hkp44_8b0cc2fa-0371-42f4-8f82-9392f6d19485/manager/0.log"
Oct 30 23:22:41.746914 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:41.746881    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-l5gbl_43439f21-b41f-4d69-bcbc-8677b0082845/manager/0.log"
Oct 30 23:22:42.147817 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:42.147769    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9nhzz_a8227528-1913-4c13-ae2f-fa15e9e27876/manager/0.log"
Oct 30 23:22:42.918430 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:42.918396    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm_26534606-c9fc-4de9-a421-8c49e98daeef/istio-proxy/0.log"
Oct 30 23:22:43.730622 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:43.730589    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-24vbj_9c8d4001-3885-49c0-84b2-1bcb032aa3d5/istio-proxy/0.log"
Oct 30 23:22:44.545715 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:44.545682    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-h8mv5_20b2d85e-38be-49c2-8ef6-31644a816dce/istio-proxy/0.log"
Oct 30 23:22:44.745028 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:44.745002    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hcq62_79c1511e-3651-415c-a1f7-96b1d89b447a/istio-proxy/0.log"
Oct 30 23:22:45.334061 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:45.334025    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-tbk5w_e50d3c0f-8e93-4e9b-9983-761f99918584/istio-proxy/0.log"
Oct 30 23:22:45.725857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:45.725821    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q_4718e016-9e2a-484a-ac25-b27ddd2b1f62/istio-proxy/0.log"
Oct 30 23:22:45.900922 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:45.900891    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-s6mwr_d9267c00-aab9-4732-98c5-b5fc7eea8cf8/payload-pre-processing/0.log"
Oct 30 23:22:47.475232 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:47.475187    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xdzkp_90d4ed54-8bcf-442b-82c4-cbae47556237/main/0.log"
Oct 30 23:22:47.858395 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:47.858357    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4_17847819-d7b1-4946-8a3c-326d50cdd08e/main/0.log"
Oct 30 23:22:49.299914 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:49.299882    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd789d65d-jvcf8_cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec/authorino/0.log"
Oct 30 23:22:50.027015 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:50.026980    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"]
Oct 30 23:22:50.036141 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:50.036114    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqlpq"]
Oct 30 23:22:50.095769 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:50.095736    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ede7f6-f043-4dee-9191-7492fe5bf7e3" path="/var/lib/kubelet/pods/66ede7f6-f043-4dee-9191-7492fe5bf7e3/volumes"
Oct 30 23:22:58.156535 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:22:58.156501    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d5c88fbff-mvdht_9e4af176-ced2-4a38-93f8-cbe8130eaef2/manager/0.log"
Oct 30 23:23:00.510847 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:00.510803    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fb84f_e7b6f703-1bff-4312-b59f-0d9188b00754/revoke-keys/0.log"
Oct 30 23:23:00.697099 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:00.697068    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-12e97699-lqd72_0e301391-4cba-4b60-8961-7ac668e7b756/revoke-keys/0.log"
Oct 30 23:23:00.881340 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:00.881307    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-45gj7_271aabed-e533-4934-956c-747525e2900b/revoke-keys/0.log"
Oct 30 23:23:01.083500 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:01.083473    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-pz46x_3e395fdb-367e-47d9-ae73-bc7d973dfc76/revoke-keys/0.log"
Oct 30 23:23:01.278360 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:01.278247    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-kh565_3b9024ab-69ce-4324-b364-61cc3f20745b/revoke-keys/0.log"
Oct 30 23:23:02.613997 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:02.613966    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd789d65d-jvcf8_cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec/authorino/0.log"
Oct 30 23:23:02.880284 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:02.880180    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hkp44_8b0cc2fa-0371-42f4-8f82-9392f6d19485/manager/0.log"
Oct 30 23:23:03.479650 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:03.479604    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-l5gbl_43439f21-b41f-4d69-bcbc-8677b0082845/manager/0.log"
Oct 30 23:23:03.905189 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:03.905162    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9nhzz_a8227528-1913-4c13-ae2f-fa15e9e27876/manager/0.log"
Oct 30 23:23:04.482025 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:04.481996    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm_26534606-c9fc-4de9-a421-8c49e98daeef/istio-proxy/0.log"
Oct 30 23:23:05.224188 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:05.224154    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-24vbj_9c8d4001-3885-49c0-84b2-1bcb032aa3d5/istio-proxy/0.log"
Oct 30 23:23:06.018451 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:06.018421    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-h8mv5_20b2d85e-38be-49c2-8ef6-31644a816dce/istio-proxy/0.log"
Oct 30 23:23:06.215034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:06.214997    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hcq62_79c1511e-3651-415c-a1f7-96b1d89b447a/istio-proxy/0.log"
Oct 30 23:23:06.713578 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:06.713545    4581 scope.go:122] "RemoveContainer" containerID="83300ad8c1add5a9bcd1c855069bda3712ace5300a28785b197acb5e5bd642d7"
Oct 30 23:23:06.722873 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:06.722853    4581 scope.go:122] "RemoveContainer" containerID="9502eeb28774eb72301c4c047ab644ec8a3d3c285f25af71d0bf3db546c9ce8a"
Oct 30 23:23:06.780924 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:06.780883    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-tbk5w_e50d3c0f-8e93-4e9b-9983-761f99918584/istio-proxy/0.log"
Oct 30 23:23:07.168643 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:07.168611    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q_4718e016-9e2a-484a-ac25-b27ddd2b1f62/istio-proxy/0.log"
Oct 30 23:23:07.352151 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:07.352121    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-s6mwr_d9267c00-aab9-4732-98c5-b5fc7eea8cf8/payload-pre-processing/0.log"
Oct 30 23:23:08.682301 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:08.682269    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xdzkp_90d4ed54-8bcf-442b-82c4-cbae47556237/main/0.log"
Oct 30 23:23:09.088342 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:09.088265    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s52p4_17847819-d7b1-4946-8a3c-326d50cdd08e/main/0.log"
Oct 30 23:23:10.042904 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:10.042870    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"]
Oct 30 23:23:10.051792 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:10.051762    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45gj7"]
Oct 30 23:23:10.096870 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:10.096836    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271aabed-e533-4934-956c-747525e2900b" path="/var/lib/kubelet/pods/271aabed-e533-4934-956c-747525e2900b/volumes"
Oct 30 23:23:10.815200 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:23:10.815157    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:23:32.143603 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:32.143569    4581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"]
Oct 30 23:23:32.156202 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:32.156164    4581 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e97699-lqd72"]
Oct 30 23:23:34.096034 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:23:34.095999    4581 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e301391-4cba-4b60-8961-7ac668e7b756" path="/var/lib/kubelet/pods/0e301391-4cba-4b60-8961-7ac668e7b756/volumes"
Oct 30 23:23:40.816351 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:23:40.816310    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:24:06.750475 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:06.750397    4581 scope.go:122] "RemoveContainer" containerID="099de6940bed3e0ca6765752c6422b463ea3a5e272d30a9febc0a306baf15a3c"
Oct 30 23:24:06.760000 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:06.759977    4581 scope.go:122] "RemoveContainer" containerID="9dbca3fe12ccd651bfa99c18c72de421a0dcc22f3a71318628368a32db447f8f"
Oct 30 23:24:10.816286 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:24:10.816242    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:24:32.588742 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:32.588698    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lcd4g_099dbe30-a9b3-4166-b289-7fc8e25ff414/global-pull-secret-syncer/0.log"
Oct 30 23:24:33.826251 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:33.826204    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vgcnn_c7665c55-0b43-4318-9bad-342fce7a359d/konnectivity-agent/0.log"
Oct 30 23:24:34.115358 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:34.115325    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-132.ec2.internal_3f5710f91ddba841cbff091b973dc1da/haproxy/0.log"
Oct 30 23:24:40.322559 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:40.322518    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd789d65d-jvcf8_cf6b4c61-e0c0-48ea-9eec-d362e6fec6ec/authorino/0.log"
Oct 30 23:24:40.429555 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:40.429526    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hkp44_8b0cc2fa-0371-42f4-8f82-9392f6d19485/manager/0.log"
Oct 30 23:24:40.817792 ip-10-0-135-132 kubenswrapper[4581]: E1030 23:24:40.817756    4581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd57c" containerName="extractor"
Oct 30 23:24:40.828811 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:40.828786    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-l5gbl_43439f21-b41f-4d69-bcbc-8677b0082845/manager/0.log"
Oct 30 23:24:41.096011 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:41.095939    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9nhzz_a8227528-1913-4c13-ae2f-fa15e9e27876/manager/0.log"
Oct 30 23:24:42.068817 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.068775    4581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"]
Oct 30 23:24:42.069440 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.069417    4581 cpu_manager.go:414] "containerMap: removing container" podUID="3e395fdb-367e-47d9-ae73-bc7d973dfc76" containerName="revoke-keys"
Oct 30 23:24:42.069549 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.069443    4581 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e395fdb-367e-47d9-ae73-bc7d973dfc76" containerName="revoke-keys"
Oct 30 23:24:42.069549 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.069543    4581 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e395fdb-367e-47d9-ae73-bc7d973dfc76" containerName="revoke-keys"
Oct 30 23:24:42.072857 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.072833    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.085035 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.085012    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qktfv\"/\"kube-root-ca.crt\""
Oct 30 23:24:42.095585 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.095547    4581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qktfv\"/\"default-dockercfg-qgg56\""
Oct 30 23:24:42.095755 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.095592    4581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qktfv\"/\"openshift-service-ca.crt\""
Oct 30 23:24:42.099722 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.099690    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"]
Oct 30 23:24:42.113312 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.113278    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-podres\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.113501 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.113319    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhvcb\" (UniqueName: \"kubernetes.io/projected/0a29913b-5537-45a5-8105-0b455220b12b-kube-api-access-nhvcb\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.113501 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.113446    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-proc\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.113653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.113522    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-sys\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.113653 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.113625    4581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-lib-modules\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.215798    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-sys\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.215928    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-lib-modules\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.215988    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-podres\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.216027    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhvcb\" (UniqueName: \"kubernetes.io/projected/0a29913b-5537-45a5-8105-0b455220b12b-kube-api-access-nhvcb\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216272 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.216114    4581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-proc\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216772 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.216743    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-proc\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216929 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.216783    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-lib-modules\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216929 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.216842    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-podres\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.216929 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.216880    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a29913b-5537-45a5-8105-0b455220b12b-sys\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.242094 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.242060    4581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhvcb\" (UniqueName: \"kubernetes.io/projected/0a29913b-5537-45a5-8105-0b455220b12b-kube-api-access-nhvcb\") pod \"perf-node-gather-daemonset-7b6cg\" (UID: \"0a29913b-5537-45a5-8105-0b455220b12b\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.385511 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.385478    4581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.688786 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.688702    4581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"]
Oct 30 23:24:42.691917 ip-10-0-135-132 kubenswrapper[4581]: W1030 23:24:42.691885    4581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0a29913b_5537_45a5_8105_0b455220b12b.slice/crio-8666658b01efe2f5b5de4671fb63152033be2eac9b46b4844c733f1fdcf51959 WatchSource:0}: Error finding container 8666658b01efe2f5b5de4671fb63152033be2eac9b46b4844c733f1fdcf51959: Status 404 returned error can't find the container with id 8666658b01efe2f5b5de4671fb63152033be2eac9b46b4844c733f1fdcf51959
Oct 30 23:24:42.877252 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.877186    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg" event={"ID":"0a29913b-5537-45a5-8105-0b455220b12b","Type":"ContainerStarted","Data":"9a19b9e8dc3fba9202f84e51f596d1a48aa3584cf736d5682fc0f4fb7284a5cc"}
Oct 30 23:24:42.877446 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.877278    4581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg" event={"ID":"0a29913b-5537-45a5-8105-0b455220b12b","Type":"ContainerStarted","Data":"8666658b01efe2f5b5de4671fb63152033be2eac9b46b4844c733f1fdcf51959"}
Oct 30 23:24:42.877446 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.877398    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:42.909385 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:42.909331    4581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg" podStartSLOduration=0.90931898 podStartE2EDuration="909.31898ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:42.90914282 +0000 UTC m=+4666.948081698" watchObservedRunningTime="2026-10-30 23:24:42.90931898 +0000 UTC m=+4666.948257866"
Oct 30 23:24:44.978083 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:44.978054    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nfjcp_8d14e9a0-d597-4d44-a61e-fb267d348a54/node-exporter/0.log"
Oct 30 23:24:45.102380 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:45.102349    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nfjcp_8d14e9a0-d597-4d44-a61e-fb267d348a54/kube-rbac-proxy/0.log"
Oct 30 23:24:45.228273 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:45.228200    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nfjcp_8d14e9a0-d597-4d44-a61e-fb267d348a54/init-textfile/0.log"
Oct 30 23:24:47.277801 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:47.277769    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fd7577f5-njz8x_6551a999-524e-4039-9d02-043975af6d73/thanos-query/0.log"
Oct 30 23:24:47.403563 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:47.403534    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fd7577f5-njz8x_6551a999-524e-4039-9d02-043975af6d73/kube-rbac-proxy-web/0.log"
Oct 30 23:24:47.529704 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:47.529617    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fd7577f5-njz8x_6551a999-524e-4039-9d02-043975af6d73/kube-rbac-proxy/0.log"
Oct 30 23:24:47.654709 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:47.654677    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fd7577f5-njz8x_6551a999-524e-4039-9d02-043975af6d73/prom-label-proxy/0.log"
Oct 30 23:24:47.779100 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:47.779070    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fd7577f5-njz8x_6551a999-524e-4039-9d02-043975af6d73/kube-rbac-proxy-rules/0.log"
Oct 30 23:24:47.904395 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:47.904364    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fd7577f5-njz8x_6551a999-524e-4039-9d02-043975af6d73/kube-rbac-proxy-metrics/0.log"
Oct 30 23:24:48.895645 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:48.895614    4581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-7b6cg"
Oct 30 23:24:51.441682 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:51.441639    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6676d75c9-mh2kk_7767cc7c-dedd-4ee0-b153-9076a61a563d/console/0.log"
Oct 30 23:24:51.593373 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:51.593344    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-plmfm_3b303815-9277-4c80-9f8b-14fed36152b3/download-server/0.log"
Oct 30 23:24:55.069197 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:55.069147    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm_26534606-c9fc-4de9-a421-8c49e98daeef/istio-proxy/0.log"
Oct 30 23:24:55.778374 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:55.778339    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tjbxb_ec7d111b-c9ab-455f-acdd-a016d4f48531/dns/0.log"
Oct 30 23:24:55.900687 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:55.900659    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tjbxb_ec7d111b-c9ab-455f-acdd-a016d4f48531/kube-rbac-proxy/0.log"
Oct 30 23:24:56.343961 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:56.343927    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ndn7k_df124a61-d687-4ef9-a732-2d5def6ef79c/dns-node-resolver/0.log"
Oct 30 23:24:57.588327 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:57.588298    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6897644698-7z8kx_ad9e8db5-c50a-4e8f-be5e-11c41d078f13/registry/0.log"
Oct 30 23:24:57.713997 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:57.713966    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-24vbj_9c8d4001-3885-49c0-84b2-1bcb032aa3d5/istio-proxy/0.log"
Oct 30 23:24:57.746234 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:57.746190    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6zmgp_91c86d61-12bf-4446-b98a-f9e571ad5ffb/node-ca/0.log"
Oct 30 23:24:57.949815 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:57.949790    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-h8mv5_20b2d85e-38be-49c2-8ef6-31644a816dce/istio-proxy/0.log"
Oct 30 23:24:58.022164 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:58.022128    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hcq62_79c1511e-3651-415c-a1f7-96b1d89b447a/istio-proxy/0.log"
Oct 30 23:24:58.202306 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:58.202230    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-tbk5w_e50d3c0f-8e93-4e9b-9983-761f99918584/istio-proxy/0.log"
Oct 30 23:24:58.318430 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:58.318397    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-scp7q_4718e016-9e2a-484a-ac25-b27ddd2b1f62/istio-proxy/0.log"
Oct 30 23:24:58.378392 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:58.378367    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-s6mwr_d9267c00-aab9-4732-98c5-b5fc7eea8cf8/payload-pre-processing/0.log"
Oct 30 23:24:59.986189 ip-10-0-135-132 kubenswrapper[4581]: I1030 23:24:59.986153    4581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5kw5cm_26534606-c9fc-4de9-a421-8c49e98daeef/istio-proxy/0.log"